Geek asks and answers

Set nonzero DataFrame values ​​conditionally

I have a dataframe:

     0         1         2   3         4  y
35 NaN       NaN       NaN NaN  0.342153  0
40 NaN  0.326323       NaN NaN       NaN  0
43 NaN       NaN  0.290126 NaN       NaN  0
49 NaN  0.326323       NaN NaN       NaN  0
50 NaN  0.391147       NaN NaN       NaN  1

And the code to create it:

import pandas as pd
import numpy as np

nan = np.nan

df = pd.DataFrame(
{0L: {35: nan, 40: nan, 43: nan, 49: nan, 50: nan},
 1L: {35: nan,
  40: 0.32632316859446198,
  43: nan,
  49: 0.32632316859446198,
  50: 0.39114724480578139},
 2L: {35: nan, 40: nan, 43: 0.29012581014105987, 49: nan, 50: nan},
 3L: {35: nan, 40: nan, 43: nan, 49: nan, 50: nan},
 4L: {35: 0.34215328467153283, 40: nan, 43: nan, 49: nan, 50: nan},
 'y': {35: 0, 40: 0, 43: 0, 49: 0, 50: 1}})

I need to assign a value to each column using the following pseudocode:

column = 1 if column > threshold else 0 where column != NaN

I tried using fancy indexing to do it like this:

df.ix[df[1].notnull(),1] = 1; df

     0   1         2   3         4  y
35 NaN NaN       NaN NaN  0.342153  0
40 NaN   1       NaN NaN       NaN  0
43 NaN NaN  0.290126 NaN       NaN  0
49 NaN   1       NaN NaN       NaN  0
50 NaN   1       NaN NaN       NaN  1

But A) I'm not sure how to apply conditional logic, and B) I have to apply logic for each column iteratively, and not for the entire data frame.

Question:

How can I apply conditional logic to non-zero values ​​of a data frame while preserving the validity of other fields?

+5
source share
3 answers
# you need this because your y column is an int64 (otherwise this the next step
# will throw an exception), on the to fix list in 0.11-dev though
In [71]: df = orig_df.astype('float64')

# use boolean indexing!
# NaN are automatically excluded
In [72]: df[df>0.3] = 1 ; df[df<=0.3] = 0

In [73]: df
Out[73]: 
     0   1         2   3   4  y
35 NaN NaN       NaN NaN   1  0
40 NaN   1       NaN NaN NaN  0
43 NaN NaN         0 NaN NaN  0
49 NaN   1       NaN NaN NaN  0
50 NaN   1       NaN NaN NaN  1
+5
source

You can use applymapas it seems like you really need an operation using an element:

>>> df.applymap(lambda x: x if pd.isnull(x) else (1 if x > 0.3 else 0))
     0   1   2   3   4  y
35 NaN NaN NaN NaN   1  0
40 NaN   1 NaN NaN NaN  0
43 NaN NaN   0 NaN NaN  0
49 NaN   1 NaN NaN NaN  0
50 NaN   1 NaN NaN NaN  1

():

>>> (df > 0.3) * 1 + df * 0
     0   1   2   3   4  y
35 NaN NaN NaN NaN   1  0
40 NaN   1 NaN NaN NaN  0
43 NaN NaN   0 NaN NaN  0
49 NaN   1 NaN NaN NaN  0
50 NaN   1 NaN NaN NaN  1
+3

, , iterrows(). pandas

I was not 100% sure what you were trying to do using pseudo-code, but the following would change a new series (which can be assigned to a data frame).

updated = df['data']
for index, row in df.iterrows():
    if ((not pd.isnull(df[index])) and df[index] > threshold) :
        updated[index] = 1.0
    else:
        updated[index] = 0.0
df['data'] = updated

This will most likely give you a warning SettingWithCopyWarningabout the assignment in the iterator, so use it with caution.

+1
source

More articles:


All Articles