Styjun

Warranty on lock damaged during attempted theft

Driving distance between O'Hare (ORD) and Downtown Chicago is 20.5 mi and 1H 17min?

How prevalent is the sound /ʑ:/ in Russian?

Simplify makes Mathematica forget that a matrix is Hermitian

Will an administrator exceed the 401K limit?

Interview question: If correlation doesn't imply causation, how do you detect causation?

Is it possible to use gases instead of liquids as fuel in a rocket engine?

What is the 12th Russian exception verb?

A randomized encryption program

How to implement the five star notation in Latex

If Space Shuttle flies "like a brick", why does it need the wings?

Paying to leave without notice in at-will employment state

Coordinate entry in QGIS 3

How can I find replacement part for Mongoose scooter?

Bo Derek in texbook.tex?

Why do airline tickets have titles in addition to names?

Seeing the ground from the shuttle before launch?

Identify the Eeveelutions

How do I experimentally measure the surface area of a rock?

Is it possible to get reverse life insurance?

Average Two Letters

How to determine the degree to which diminished seventh chords and half-diminished seventh chords belong in major scales?

What is the meaning of 奥手 here?

When the direction of a movement changes, is the object at rest at some time?

what is the best way to deal with a lot of dataset of different type (case classes)

How to define partitioning of DataFrame?Parallelize GZip file processing SparkHow to read a parquet file with lots of columns to a Dataset without a custom case class?Scala 2.11 & Spark 2.0.0 Create dynamically case class to encode DatasetHow to call Spark UDF on Datasets of case classes?Spark SQL - Read csv into Dataset[T] where T is a case class of Option[BigDecimal] fieldAvro Schema vs. Scala Case Class for Spark DatasetsStream files from HDFS using Apache Spark SteamingHow to create merge-sorted partitions in Sparkspark convert dataframe to dataset using case class with option fields

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

0

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Are they all different schema ?
  
  – howie
  Mar 28 at 22:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes 50 + different parquet with different schema
  
  – rio
  Apr 4 at 21:34
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  usually when I have a few I use a def and return a tuple ex ``` val (ds1,ds2, ds3) = getDS ```
  
  – rio
  Apr 4 at 21:35
  
  

  
  
  
  

add a comment
 | 

0

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Are they all different schema ?
  
  – howie
  Mar 28 at 22:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes 50 + different parquet with different schema
  
  – rio
  Apr 4 at 21:34
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  usually when I have a few I use a def and return a tuple ex ``` val (ds1,ds2, ds3) = getDS ```
  
  – rio
  Apr 4 at 21:35
  
  

  
  
  
  

add a comment
 | 

I have like 50+ different case classes and each one is bound to a different parquet source

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

how to handle them in case of need to process them?
I cannot use

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

and I would like to avoid having so many vals defined

apache-spark apache-spark-sql apache-spark-dataset

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

val cc1ds = spark.read("/hdfs/input1").as[cc1]
val cc2ds = spark.read("/hdfs/input1").as[cc2]
...
val cc50ds = spark.read("/hdfs/input1").as[cc50]

val res:(dataset[cc1], ... dataset[cc50]) = (cc1ds, ... cc50ds)

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

share|improve this question

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

edited Mar 28 at 22:08

Roberto Congiu

3,82511 gold badge1616 silver badges2929 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges

asked Mar 28 at 21:57

riorio

30533 silver badges1313 bronze badges