Styjun

What was the profession 芸者 (female entertainer) called in Germany?

I make billions (#6)

Novel with societal breakdown and spaceship passengers marooned on a planet covered with a city

How to configure apt in Debian Buster after release

This LM317 diagram doesn't make any sense to me

Is it ok for parents to kiss and romance with each other while their 2- to 8-year-old child watches?

Why the Cauchy Distribution is so useful?

Hail hit my roof. Do I need to replace it?

What are the effects of abstaining from eating a certain flavor?

Wires do not connect in Circuitikz

What is a writing material that persists forever or for a long time?

Passwordless authentication - how and when to invalidate a login code

Did depressed people far more accurately estimate how many monsters they killed in a video game?

Why AI became applicable only after Nvidia's chips were available?

Users forgetting to regenerate PDF before sending it

How to evaluate the performance of open source solver?

Good sources on developing mathematical models

Interpretation of non-significant results as "trends"

When do flights get cancelled due to fog?

What does the multimeter dial do internally?

Run Bash scripts in folder all at the same time

Can a landlord force all residents to use the landlord's in-house debit card accounts?

How does one acquire an undead eyeball encased in a gem?

Can you cast the Shape Water spell without an existing obvious pool of water?

Multiple Filters in Pyspark

PySpark: Filter a DataFrame using conditionPySpark dataframe filter on multiple columnsMultiple Filtering in PySparkPySpark RDD Filter with “not in” for multiple valuespyspark dataframe filtering on multiple columnsWriting a dataframe to disk taking an unrealistically long time in Pyspark (Spark 2.1.1)Pyspark windows on last 30 days on subset of dataPySpark multiple filters not workingPyspark compound filter, multiple conditionsPySpark filtering gives inconsistent behavior

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

Need to filter the Data using multiple conditions based on record codes and date of services and count the distinct values based on the col1, col2,col3.

Having issue with the Pyspark parameters resolving during execution and returning no records.

from_dt = 01-01-2018
end_dt= 12-31-2018

df.filter((trim(df.code) =='AB') | (trim(df.code) =='CD') | (trim(df.code) =='F')).filter("from_dt >= '$0' & end_dt <= $1'").select("col1","col2","col3").distinct().count()

pyspark

share|improve this question

edited Mar 26 at 0:31

Kumar

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Try keeping the entire contents of the filter in braces, also are you sure there will be some rows returned after applying these filters ?
  
  – Vaibhav
  Mar 26 at 10:27
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Add sample input and expected output with output you are getting from your experiment
  
  – Rakesh Kumar
  Mar 27 at 2:30
  

  
  
  
  

add a comment | 

0

Need to filter the Data using multiple conditions based on record codes and date of services and count the distinct values based on the col1, col2,col3.

Having issue with the Pyspark parameters resolving during execution and returning no records.

from_dt = 01-01-2018
end_dt= 12-31-2018

df.filter((trim(df.code) =='AB') | (trim(df.code) =='CD') | (trim(df.code) =='F')).filter("from_dt >= '$0' & end_dt <= $1'").select("col1","col2","col3").distinct().count()

pyspark

share|improve this question

edited Mar 26 at 0:31

Kumar

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Try keeping the entire contents of the filter in braces, also are you sure there will be some rows returned after applying these filters ?
  
  – Vaibhav
  Mar 26 at 10:27
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Add sample input and expected output with output you are getting from your experiment
  
  – Rakesh Kumar
  Mar 27 at 2:30
  

  
  
  
  

add a comment | 

Need to filter the Data using multiple conditions based on record codes and date of services and count the distinct values based on the col1, col2,col3.

Having issue with the Pyspark parameters resolving during execution and returning no records.

from_dt = 01-01-2018
end_dt= 12-31-2018

df.filter((trim(df.code) =='AB') | (trim(df.code) =='CD') | (trim(df.code) =='F')).filter("from_dt >= '$0' & end_dt <= $1'").select("col1","col2","col3").distinct().count()

pyspark

share|improve this question

edited Mar 26 at 0:31

Kumar

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

from_dt = 01-01-2018
end_dt= 12-31-2018

df.filter((trim(df.code) =='AB') | (trim(df.code) =='CD') | (trim(df.code) =='F')).filter("from_dt >= '$0' & end_dt <= $1'").select("col1","col2","col3").distinct().count()

share|improve this question

edited Mar 26 at 0:31

Kumar

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

share|improve this question

edited Mar 26 at 0:31

Kumar

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

asked Mar 25 at 22:44

KumarKumar

122 bronze badges

asked Mar 25 at 22:44

KumarKumar

122 bronze badges