Styjun

Is Fourier series a sampled version of Fourier transform?

Solving pricing problem heuristically in column generation algorithm for VRP

Minimum population for language survival

How do I answer an interview question about not meeting deadlines?

Doesn't the speed of light limit imply the same electron can be annihilated twice?

Why won't the Republicans use a superdelegate system like the DNC in their nomination process?

What is the question mark?

Units of measurement, especially length, when body parts vary in size among races

Mind ya, it's Homophones Everywhere!

RAII wrapper for SQLite transactions

Duplicate and slide edge (rip from boundary)

Is the Microsoft recommendation to use C# properties applicable to game development?

What should we do with manuals from the 80s?

Is there any official ruling on how characters go from 0th to 1st level in a class?

What was the intention with the Commodore 128?

Good way to stop electrolyte tabs from turning into powder?

How can I find an old paper when the usual methods fail?

Does the Haste spell's hasted action allow you to make multiple unarmed strikes? Or none at all?

Short comic about alien explorers visiting an abandoned world with giant statues that turn out to be alive but move very slowly

What are these panels underneath the wing root of a A380?

Setting up a Mathematical Institute of Refereeing?

Does Reckless Attack work with Multiattack when wild shaped?

Weird resistor with dots around it on the schematic

What would cause a nuclear power plant to break down after 2000 years, but not sooner?

How to filter column on values in list in pyspark?

How to filter dataframe to get rows which have column value IN a user-defined setUpdating a dataframe column in sparkColumn filtering in PySparkhow to change a Dataframe column from String type to Double type in pysparkPySpark: How to fillna values in dataframe for specific columns?How to apply function to each row of specified column of PySpark DataFrameAdd a priority column in PySpark dataframeSparkSQL “CASE WHEN THEN” with two table columns in pysparkPyspark -Filtering the rows based on languageApply describe with filter on column specific in pysparkHow to comparing pair of columns using udf in pyspark?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

4

I have a dataframe rawdata on which i have to apply filter condition on column X with values CB,CI and CR. So I used the below code:

df = dfRawData.filter(col("X").between("CB","CI","CR"))

But I am getting the following error:

between() takes exactly 3 arguments (4 given)

Please let me know how I can resolve this issue.

apache-spark pyspark apache-spark-sql spark-dataframe pyspark-sql

share|improve this question

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

add a comment | 

4

I have a dataframe rawdata on which i have to apply filter condition on column X with values CB,CI and CR. So I used the below code:

df = dfRawData.filter(col("X").between("CB","CI","CR"))

But I am getting the following error:

between() takes exactly 3 arguments (4 given)

Please let me know how I can resolve this issue.

apache-spark pyspark apache-spark-sql spark-dataframe pyspark-sql

share|improve this question

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

add a comment | 

I have a dataframe rawdata on which i have to apply filter condition on column X with values CB,CI and CR. So I used the below code:

df = dfRawData.filter(col("X").between("CB","CI","CR"))

But I am getting the following error:

between() takes exactly 3 arguments (4 given)

Please let me know how I can resolve this issue.

apache-spark pyspark apache-spark-sql spark-dataframe pyspark-sql

share|improve this question

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

df = dfRawData.filter(col("X").between("CB","CI","CR"))

share|improve this question

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

share|improve this question

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

edited Oct 18 '17 at 1:36

Shaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

asked Oct 12 '17 at 10:30

LKALKA

5522 gold badges22 silver badges99 bronze badges

1 Answer
1

active

oldest

votes

14

between is used to check if the value is between two values, the input is a lower bound and an upper bound. It can not be used to check if a column value is in a list. To do that, use isin:

df = dfRawData.where(col("X").isin("CB", "CI", "CR"))

share|improve this answer

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f46707339%2fhow-to-filter-column-on-values-in-list-in-pyspark%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

14

between is used to check if the value is between two values, the input is a lower bound and an upper bound. It can not be used to check if a column value is in a list. To do that, use isin:

df = dfRawData.where(col("X").isin("CB", "CI", "CR"))

share|improve this answer

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

add a comment | 

14

between is used to check if the value is between two values, the input is a lower bound and an upper bound. It can not be used to check if a column value is in a list. To do that, use isin:

df = dfRawData.where(col("X").isin("CB", "CI", "CR"))

share|improve this answer

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

add a comment | 

between is used to check if the value is between two values, the input is a lower bound and an upper bound. It can not be used to check if a column value is in a list. To do that, use isin:

df = dfRawData.where(col("X").isin("CB", "CI", "CR"))

share|improve this answer

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

df = dfRawData.where(col("X").isin("CB", "CI", "CR"))

share|improve this answer

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges

answered Oct 12 '17 at 10:54

ShaidoShaido

15.1k1212 gold badges3333 silver badges4646 bronze badges