Styjun

Modified stem cells as a resuscitation serum after death by cyanide poisoning?

RPMs too high on freeway?

Should I tell an editor that I believe an article I'm reviewing is not good enough for the journal?

Why are session states synchronized with high availability?

What is the correct way for pilots to say times?

Is "montäglich" commonly used?

Isn't any conversation with the US president quid-pro-quo?

Beginner Tactics - Why Isn't This Mate?

Why are rain clouds darker?

Buddha Hinduism

Does removing finitely many points from an open set yield an open set?

Translate 主播 in this context

Showing a point on a plot as a circle rather than a disk

Did Ohio pass a law granting students the right to give scientifically wrong answers consistent with their religious beliefs?

Why do we use the particular magnitude of a force to calculate work?

Steampunk book about a bounty hunter teen girl in London

Cage Length (Rear Derallieur) and Total Capacity

Does toddler keep hands around private parts?

Remix compile error: Expected primary expression

TSP with revenue maximization

How can I add an ammeter and/or voltmeter to my home breaker panel?

Twelve Labours #08 - Dark Horse Bookmakers

Which of these will work? HDMI to VGA or HDMI to USB?

Need help reading piano sheet music

what is the best way to deal with a lot of dataset of different type (case classes)

How to define partitioning of DataFrame?Parallelize GZip file processing SparkHow to read a parquet file with lots of columns to a Dataset without a custom case class?Scala 2.11 & Spark 2.0.0 Create dynamically case class to encode DatasetHow to call Spark UDF on Datasets of case classes?Spark SQL - Read csv into Dataset[T] where T is a case class of Option[BigDecimal] fieldAvro Schema vs. Scala Case Class for Spark DatasetsStream files from HDFS using Apache Spark SteamingHow to create merge-sorted partitions in Sparkspark convert dataframe to dataset using case class with option fields

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

0

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Are they all different schema ?
  
  – howie
  Mar 28 at 22:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes 50 + different parquet with different schema
  
  – rio
  Apr 4 at 21:34
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  usually when I have a few I use a def and return a tuple ex ``` val (ds1,ds2, ds3) = getDS ```
  
  – rio
  Apr 4 at 21:35
  
  

  
  
  
  

add a comment
 | 

0

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Are they all different schema ?
  
  – howie
  Mar 28 at 22:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes 50 + different parquet with different schema
  
  – rio
  Apr 4 at 21:34
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  usually when I have a few I use a def and return a tuple ex ``` val (ds1,ds2, ds3) = getDS ```
  
  – rio
  Apr 4 at 21:35
  
  

  
  
  
  

add a comment
 | 

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

edited Mar 28 at 22:08

Roberto Congiu

3,83511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges

asked Mar 28 at 21:57

riorio

33033 silver badges1313 bronze badges