Styjun

Has the US government provided details on plans to deal with AIDS and childhood cancer?

Went to a big 4 but got fired for underperformance in a year recently - Now every one thinks I'm pro - How to balance expectations?

Basic transistor circuit

Could flaps be raised upward to serve as spoilers / lift dumpers?

Conflict between senior and junior members

How can flights operated by the same company have such different prices when marketed by another?

Why are sugars in whole fruits not digested the same way sugars in juice are?

Security measures that could plausibly last 150+ years?

Word for giving preference to the oldest child

Skipping same old introductions

Should I put my name first or last in the team members list?

Gold Battle KoTH

How to escape forward slashes?

PI 4 screen rotation from the terminal

Why have both: BJT and FET transistors on IC output?

UX writing: When to use "we"?

Accurately recalling the key - can everyone do it?

Cross out words with TikZ: line opacity

Is the EU really banning "toxic propellants" in 2020? How is that going to work?

Return last number in sub-sequences in a list of integers

The grades of the students in a class

How do I respond appropriately to an overseas company that obtained a visa for me without hiring me?

Adding a (stair/baby) gate without facing walls

Backpacking with incontinence

Pandas filter dataframe based on condition for the first n rows

Add one row to pandas DataFrameSelecting multiple columns in a pandas dataframeAdding new column to existing DataFrame in Python pandasDelete column from pandas DataFrameDelete rows from a pandas DataFrame based on a conditional expression involving len(string) giving KeyErrorHow do I get the row count of a pandas DataFrame?How to iterate over rows in a DataFrame in Pandas?Select rows from a DataFrame based on values in a column in pandasDeleting DataFrame row in Pandas based on column valueGet list from pandas DataFrame column headers

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

4

1

I have a dataframe of shape [600 000, 19]. I want to filter the first 100 000 rows based on one condition, the next 300 000 based on another condition, and a 3rd condition for the last rows. I was wondering how this can be done.

Currently, I split the data frame into 3 segments and apply their respective conditions. Then, I re-concatenate the data frame. Is there a better way?

Example: Filter first 100 000 rows based on any value less than 5. For second 300 000 rows, I dont want any values greater than 40, etc.

python pandas filtering conditional-statements

share|improve this question

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  reset index into a column then you can incorporate the row sequence number into your condition
  
  – adrtam
  Mar 26 at 23:51
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Try creating three masks using .iloc and then appply the three masks at once using 'or' ( | ) to the dataframe.
  
  – run-out
  Mar 27 at 0:23
  

  
  
  
  

add a comment | 

4

1

I have a dataframe of shape [600 000, 19]. I want to filter the first 100 000 rows based on one condition, the next 300 000 based on another condition, and a 3rd condition for the last rows. I was wondering how this can be done.

Currently, I split the data frame into 3 segments and apply their respective conditions. Then, I re-concatenate the data frame. Is there a better way?

Example: Filter first 100 000 rows based on any value less than 5. For second 300 000 rows, I dont want any values greater than 40, etc.

python pandas filtering conditional-statements

share|improve this question

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  reset index into a column then you can incorporate the row sequence number into your condition
  
  – adrtam
  Mar 26 at 23:51
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Try creating three masks using .iloc and then appply the three masks at once using 'or' ( | ) to the dataframe.
  
  – run-out
  Mar 27 at 0:23
  

  
  
  
  

add a comment | 

I have a dataframe of shape [600 000, 19]. I want to filter the first 100 000 rows based on one condition, the next 300 000 based on another condition, and a 3rd condition for the last rows. I was wondering how this can be done.

Currently, I split the data frame into 3 segments and apply their respective conditions. Then, I re-concatenate the data frame. Is there a better way?

Example: Filter first 100 000 rows based on any value less than 5. For second 300 000 rows, I dont want any values greater than 40, etc.

python pandas filtering conditional-statements

share|improve this question

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

share|improve this question

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

share|improve this question

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

asked Mar 26 at 23:40

Rui NianRui Nian

66944 silver badges1313 bronze badges

2 Answers
2

active

oldest

votes

2

You can try the following approach:

import pandas as pd

sample = pd.DataFrame('x' : pd.np.arange(100),
 'colname': pd.np.arange(100))
conditions = [('index < 5', 'colname < 3'), 
 ('index > 50', 'index < 100', 'colname < 55')]
sample.query('|'.join(map(lambda x: '&'.join(x), conditions)))

share|improve this answer

edited Mar 27 at 0:34

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This definitely was able to accomplish the task without splitting and reconcat'ing. I also learned a few other things about Pandas.
  
  – Rui Nian
  Mar 27 at 16:40
  

  
  
  
  

add a comment | 

1

On approach would be to use dataframe index slicing with pd.concat to build complete boolean series:

import numpy as np
import pandas as pd
np.random.seed(0)
df=pd.DataFrame(np.random.randint(0,50,60))

df[pd.concat([df.iloc[:10] > 10, df[11:40] < 30, df[41:] % 2 == 0])]

Where first 10 records filters less than 10, next 30 values filters greater than 30, and last values check for even numbers.

Then you can use dropna to remove all the NaN values

Output:

 0
0 44.0
1 47.0
2 NaN
3 NaN
4 NaN
5 39.0
6 NaN
7 19.0
8 21.0
9 36.0
10 NaN
11 6.0
12 24.0
13 24.0
14 12.0
15 1.0
16 NaN
17 NaN
18 23.0
19 NaN
20 24.0
21 17.0
22 NaN
23 25.0
24 13.0
25 8.0
26 9.0
27 20.0
28 16.0
29 5.0
30 15.0
31 NaN
32 0.0
33 18.0
34 NaN
35 24.0
36 NaN
37 29.0
38 19.0
39 19.0
40 NaN
41 NaN
42 32.0
43 NaN
44 NaN
45 32.0
46 NaN
47 10.0
48 NaN
49 NaN
50 NaN
51 28.0
52 34.0
53 0.0
54 0.0
55 36.0
56 NaN
57 38.0
58 40.0
59 NaN

share|improve this answer

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This is definitely a cleaner version of what I had originally, I then dropped the na's and reset the index and it works great.
  
  – Rui Nian
  Mar 27 at 16:39
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55367725%2fpandas-filter-dataframe-based-on-condition-for-the-first-n-rows%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

You can try the following approach:

import pandas as pd

sample = pd.DataFrame('x' : pd.np.arange(100),
 'colname': pd.np.arange(100))
conditions = [('index < 5', 'colname < 3'), 
 ('index > 50', 'index < 100', 'colname < 55')]
sample.query('|'.join(map(lambda x: '&'.join(x), conditions)))

share|improve this answer

edited Mar 27 at 0:34

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This definitely was able to accomplish the task without splitting and reconcat'ing. I also learned a few other things about Pandas.
  
  – Rui Nian
  Mar 27 at 16:40
  

  
  
  
  

add a comment | 

2

You can try the following approach:

import pandas as pd

sample = pd.DataFrame('x' : pd.np.arange(100),
 'colname': pd.np.arange(100))
conditions = [('index < 5', 'colname < 3'), 
 ('index > 50', 'index < 100', 'colname < 55')]
sample.query('|'.join(map(lambda x: '&'.join(x), conditions)))

share|improve this answer

edited Mar 27 at 0:34

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This definitely was able to accomplish the task without splitting and reconcat'ing. I also learned a few other things about Pandas.
  
  – Rui Nian
  Mar 27 at 16:40
  

  
  
  
  

add a comment | 

You can try the following approach:

import pandas as pd

sample = pd.DataFrame('x' : pd.np.arange(100),
 'colname': pd.np.arange(100))
conditions = [('index < 5', 'colname < 3'), 
 ('index > 50', 'index < 100', 'colname < 55')]
sample.query('|'.join(map(lambda x: '&'.join(x), conditions)))

share|improve this answer

edited Mar 27 at 0:34

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

import pandas as pd

sample = pd.DataFrame('x' : pd.np.arange(100),
 'colname': pd.np.arange(100))
conditions = [('index < 5', 'colname < 3'), 
 ('index > 50', 'index < 100', 'colname < 55')]
sample.query('|'.join(map(lambda x: '&'.join(x), conditions)))

share|improve this answer

edited Mar 27 at 0:34

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

answered Mar 27 at 0:25

bubblebubble

1,18088 silver badges1313 bronze badges

1

On approach would be to use dataframe index slicing with pd.concat to build complete boolean series:

import numpy as np
import pandas as pd
np.random.seed(0)
df=pd.DataFrame(np.random.randint(0,50,60))

df[pd.concat([df.iloc[:10] > 10, df[11:40] < 30, df[41:] % 2 == 0])]

Where first 10 records filters less than 10, next 30 values filters greater than 30, and last values check for even numbers.

Then you can use dropna to remove all the NaN values

Output:

 0
0 44.0
1 47.0
2 NaN
3 NaN
4 NaN
5 39.0
6 NaN
7 19.0
8 21.0
9 36.0
10 NaN
11 6.0
12 24.0
13 24.0
14 12.0
15 1.0
16 NaN
17 NaN
18 23.0
19 NaN
20 24.0
21 17.0
22 NaN
23 25.0
24 13.0
25 8.0
26 9.0
27 20.0
28 16.0
29 5.0
30 15.0
31 NaN
32 0.0
33 18.0
34 NaN
35 24.0
36 NaN
37 29.0
38 19.0
39 19.0
40 NaN
41 NaN
42 32.0
43 NaN
44 NaN
45 32.0
46 NaN
47 10.0
48 NaN
49 NaN
50 NaN
51 28.0
52 34.0
53 0.0
54 0.0
55 36.0
56 NaN
57 38.0
58 40.0
59 NaN

share|improve this answer

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This is definitely a cleaner version of what I had originally, I then dropped the na's and reset the index and it works great.
  
  – Rui Nian
  Mar 27 at 16:39
  

  
  
  
  

add a comment | 

1

On approach would be to use dataframe index slicing with pd.concat to build complete boolean series:

import numpy as np
import pandas as pd
np.random.seed(0)
df=pd.DataFrame(np.random.randint(0,50,60))

df[pd.concat([df.iloc[:10] > 10, df[11:40] < 30, df[41:] % 2 == 0])]

Where first 10 records filters less than 10, next 30 values filters greater than 30, and last values check for even numbers.

Then you can use dropna to remove all the NaN values

Output:

 0
0 44.0
1 47.0
2 NaN
3 NaN
4 NaN
5 39.0
6 NaN
7 19.0
8 21.0
9 36.0
10 NaN
11 6.0
12 24.0
13 24.0
14 12.0
15 1.0
16 NaN
17 NaN
18 23.0
19 NaN
20 24.0
21 17.0
22 NaN
23 25.0
24 13.0
25 8.0
26 9.0
27 20.0
28 16.0
29 5.0
30 15.0
31 NaN
32 0.0
33 18.0
34 NaN
35 24.0
36 NaN
37 29.0
38 19.0
39 19.0
40 NaN
41 NaN
42 32.0
43 NaN
44 NaN
45 32.0
46 NaN
47 10.0
48 NaN
49 NaN
50 NaN
51 28.0
52 34.0
53 0.0
54 0.0
55 36.0
56 NaN
57 38.0
58 40.0
59 NaN

share|improve this answer

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks! This is definitely a cleaner version of what I had originally, I then dropped the na's and reset the index and it works great.
  
  – Rui Nian
  Mar 27 at 16:39
  

  
  
  
  

add a comment | 

On approach would be to use dataframe index slicing with pd.concat to build complete boolean series:

import numpy as np
import pandas as pd
np.random.seed(0)
df=pd.DataFrame(np.random.randint(0,50,60))

df[pd.concat([df.iloc[:10] > 10, df[11:40] < 30, df[41:] % 2 == 0])]

Where first 10 records filters less than 10, next 30 values filters greater than 30, and last values check for even numbers.

Then you can use dropna to remove all the NaN values

Output:

 0
0 44.0
1 47.0
2 NaN
3 NaN
4 NaN
5 39.0
6 NaN
7 19.0
8 21.0
9 36.0
10 NaN
11 6.0
12 24.0
13 24.0
14 12.0
15 1.0
16 NaN
17 NaN
18 23.0
19 NaN
20 24.0
21 17.0
22 NaN
23 25.0
24 13.0
25 8.0
26 9.0
27 20.0
28 16.0
29 5.0
30 15.0
31 NaN
32 0.0
33 18.0
34 NaN
35 24.0
36 NaN
37 29.0
38 19.0
39 19.0
40 NaN
41 NaN
42 32.0
43 NaN
44 NaN
45 32.0
46 NaN
47 10.0
48 NaN
49 NaN
50 NaN
51 28.0
52 34.0
53 0.0
54 0.0
55 36.0
56 NaN
57 38.0
58 40.0
59 NaN

share|improve this answer

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

import numpy as np
import pandas as pd
np.random.seed(0)
df=pd.DataFrame(np.random.randint(0,50,60))

df[pd.concat([df.iloc[:10] > 10, df[11:40] < 30, df[41:] % 2 == 0])]

 0
0 44.0
1 47.0
2 NaN
3 NaN
4 NaN
5 39.0
6 NaN
7 19.0
8 21.0
9 36.0
10 NaN
11 6.0
12 24.0
13 24.0
14 12.0
15 1.0
16 NaN
17 NaN
18 23.0
19 NaN
20 24.0
21 17.0
22 NaN
23 25.0
24 13.0
25 8.0
26 9.0
27 20.0
28 16.0
29 5.0
30 15.0
31 NaN
32 0.0
33 18.0
34 NaN
35 24.0
36 NaN
37 29.0
38 19.0
39 19.0
40 NaN
41 NaN
42 32.0
43 NaN
44 NaN
45 32.0
46 NaN
47 10.0
48 NaN
49 NaN
50 NaN
51 28.0
52 34.0
53 0.0
54 0.0
55 36.0
56 NaN
57 38.0
58 40.0
59 NaN

share|improve this answer

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges

answered Mar 27 at 0:50

Scott BostonScott Boston

65.4k77 gold badges3939 silver badges6363 bronze badges