NoSuchElementException while using toDF from Spark / LivyGet property value from string using reflection in C#ClassNotFoundException spark-submit scalaSpark 1.6.1: creating DataFrame from RDD[Array[Error]]Spark: java.lang.UnsupportedOperationException: No Encoder found for java.time.LocalDateSpark Unit Test Fails with “ExceptionInInitializerError” with latest version of “json4s-jackson”Spark scala dataframe read and show multiline json fileI am getting error while loading my csv in spark using SQlcontextintermittent exception from a spark job: Schema for type scala.collection.Map[String,String] is not supportedSpark Structured Streaming MemoryStream + Row + Encoders issuehow to use Scala Maps with akka-kryo-serializer

How (un)safe is it to ride barefoot?

Make Gimbap cutter

What is the theme of analysis?

Why is it bad to use your whole foot in rock climbing

What do you call the action of "describing events as they happen" like sports anchors do?

Approach sick days in feedback meeting

Forgot passport for Alaska cruise (Anchorage to Vancouver)

Savage Road Signs

How can powerful telekinesis avoid violating Newton's 3rd Law?

Must I use my personal social media account for work?

How to represent jealousy in a cute way?

Why do (or did, until very recently) aircraft transponders wait to be interrogated before broadcasting beacon signals?

Oxford comma with nonessential phrases

How do I type a hyphen in iOS 12?

ISP is not hashing the password I log in with online. Should I take any action?

When to use the uncountable form of a noun?

Was planting UN flag on Moon ever discussed?

About the paper by Buekenhout, Delandtsheer, Doyen, Kleidman, Liebeck and Saxl

Why didn't the people of King's Landing riot when the Great Sept of Baelor was destroyed?

Nth term of Van Eck Sequence

The best in flight meal option for those suffering from reflux

Can you open the door or die? v2

Are the guests in Westworld forbidden to tell the hosts that they are robots?

Must CPU have a GPU if motherboard provides display port (when no separate video card)?



NoSuchElementException while using toDF from Spark / Livy


Get property value from string using reflection in C#ClassNotFoundException spark-submit scalaSpark 1.6.1: creating DataFrame from RDD[Array[Error]]Spark: java.lang.UnsupportedOperationException: No Encoder found for java.time.LocalDateSpark Unit Test Fails with “ExceptionInInitializerError” with latest version of “json4s-jackson”Spark scala dataframe read and show multiline json fileI am getting error while loading my csv in spark using SQlcontextintermittent exception from a spark job: Schema for type scala.collection.Map[String,String] is not supportedSpark Structured Streaming MemoryStream + Row + Encoders issuehow to use Scala Maps with akka-kryo-serializer






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








0















I am attempting to produce a Spark Dataframe from within Spark, which has been initialised using apache Livy.



I first noticed this issue on this more complicated hbase call:



 import spark.implicits._

 ... 

 spark.sparkContext
 .newAPIHadoopRDD(
 conf,
 classOf[TableInputFormat],
 classOf[ImmutableBytesWritable],
 classOf[Result]
 )
 .toDF()


But i found i could get the same thing to occur on a simple:



 import spark.implicits._

 ...

 val filtersDf = filters.toDF()


Where, filtersDf is just a sequence of case classes.



The common issue is the *.toDF(), however it also occurs with *.toDS(), which makes me think that the implicit resolution on import spark.implicits._ is not working. The underlying objects to be converted to dataframes do have data.



The error stack looks like it relates to runtime implicit resolution using scala runtime reflection.



Note that i have checked and both spark and the compiled code both use the same version of Scala (2.11) .



The exception i get is:



java.lang.RuntimeException: java.util.NoSuchElementException: head of empty list
scala.collection.immutable.Nil$.head(List.scala:420)
scala.collection.immutable.Nil$.head(List.scala:417)
scala.collection.immutable.List.map(List.scala:277)
scala.reflect.internal.Symbols$Symbol.parentSymbols(Symbols.scala:2117)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:301)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
scala.reflect.internal.Types$Type.member(Types.scala:600)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
scala.reflect.internal.Mirrors$RootsBase.staticPackage(Mirrors.scala:204)
scala.reflect.runtime.JavaMirrors$JavaMirror.staticPackage(JavaMirrors.scala:82)
scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:263)
scala.reflect.runtime.JavaMirrors$class.scala$reflect$runtime$JavaMirrors$$createMirror(JavaMirrors.scala:32)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:49)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:47)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.JavaMirrors$class.runtimeMirror(JavaMirrors.scala:46)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)


My working assumption is that I am missing a dependency or import and this is some kind of scala-ism.



I have yet to find any other references to this issue. Ultimately i think it is probably down to imports/dependencies but so far I can't quite see what it is. Any help greatly appreciated. I'm keen to know ways to fix the issue or alternatively to create data frames via less magical approaches than toDf().



Spark info:



Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _ / _ / _ `/ __/ '_/
 /___/ .__/_,_/_/ /_/_ version 2.3.2.0-mapr-1901
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)





scala apache-spark reflection hbase livy







share|improve this question
























  • Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

    – Apurba Pandey
    Mar 25 at 5:46











  • I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

    – ZenMasterZed
    Mar 25 at 7:12

















0















I am attempting to produce a Spark Dataframe from within Spark, which has been initialised using apache Livy.



I first noticed this issue on this more complicated hbase call:



 import spark.implicits._

 ... 

 spark.sparkContext
 .newAPIHadoopRDD(
 conf,
 classOf[TableInputFormat],
 classOf[ImmutableBytesWritable],
 classOf[Result]
 )
 .toDF()


But i found i could get the same thing to occur on a simple:



 import spark.implicits._

 ...

 val filtersDf = filters.toDF()


Where, filtersDf is just a sequence of case classes.



The common issue is the *.toDF(), however it also occurs with *.toDS(), which makes me think that the implicit resolution on import spark.implicits._ is not working. The underlying objects to be converted to dataframes do have data.



The error stack looks like it relates to runtime implicit resolution using scala runtime reflection.



Note that i have checked and both spark and the compiled code both use the same version of Scala (2.11) .



The exception i get is:



java.lang.RuntimeException: java.util.NoSuchElementException: head of empty list
scala.collection.immutable.Nil$.head(List.scala:420)
scala.collection.immutable.Nil$.head(List.scala:417)
scala.collection.immutable.List.map(List.scala:277)
scala.reflect.internal.Symbols$Symbol.parentSymbols(Symbols.scala:2117)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:301)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
scala.reflect.internal.Types$Type.member(Types.scala:600)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
scala.reflect.internal.Mirrors$RootsBase.staticPackage(Mirrors.scala:204)
scala.reflect.runtime.JavaMirrors$JavaMirror.staticPackage(JavaMirrors.scala:82)
scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:263)
scala.reflect.runtime.JavaMirrors$class.scala$reflect$runtime$JavaMirrors$$createMirror(JavaMirrors.scala:32)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:49)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:47)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.JavaMirrors$class.runtimeMirror(JavaMirrors.scala:46)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)


My working assumption is that I am missing a dependency or import and this is some kind of scala-ism.



I have yet to find any other references to this issue. Ultimately i think it is probably down to imports/dependencies but so far I can't quite see what it is. Any help greatly appreciated. I'm keen to know ways to fix the issue or alternatively to create data frames via less magical approaches than toDf().



Spark info:



Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _ / _ / _ `/ __/ '_/
 /___/ .__/_,_/_/ /_/_ version 2.3.2.0-mapr-1901
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)





scala apache-spark reflection hbase livy







share|improve this question
























  • Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

    – Apurba Pandey
    Mar 25 at 5:46











  • I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

    – ZenMasterZed
    Mar 25 at 7:12













0












0








0








I am attempting to produce a Spark Dataframe from within Spark, which has been initialised using apache Livy.



I first noticed this issue on this more complicated hbase call:



 import spark.implicits._

 ... 

 spark.sparkContext
 .newAPIHadoopRDD(
 conf,
 classOf[TableInputFormat],
 classOf[ImmutableBytesWritable],
 classOf[Result]
 )
 .toDF()


But i found i could get the same thing to occur on a simple:



 import spark.implicits._

 ...

 val filtersDf = filters.toDF()


Where, filtersDf is just a sequence of case classes.



The common issue is the *.toDF(), however it also occurs with *.toDS(), which makes me think that the implicit resolution on import spark.implicits._ is not working. The underlying objects to be converted to dataframes do have data.



The error stack looks like it relates to runtime implicit resolution using scala runtime reflection.



Note that i have checked and both spark and the compiled code both use the same version of Scala (2.11) .



The exception i get is:



java.lang.RuntimeException: java.util.NoSuchElementException: head of empty list
scala.collection.immutable.Nil$.head(List.scala:420)
scala.collection.immutable.Nil$.head(List.scala:417)
scala.collection.immutable.List.map(List.scala:277)
scala.reflect.internal.Symbols$Symbol.parentSymbols(Symbols.scala:2117)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:301)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
scala.reflect.internal.Types$Type.member(Types.scala:600)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
scala.reflect.internal.Mirrors$RootsBase.staticPackage(Mirrors.scala:204)
scala.reflect.runtime.JavaMirrors$JavaMirror.staticPackage(JavaMirrors.scala:82)
scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:263)
scala.reflect.runtime.JavaMirrors$class.scala$reflect$runtime$JavaMirrors$$createMirror(JavaMirrors.scala:32)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:49)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:47)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.JavaMirrors$class.runtimeMirror(JavaMirrors.scala:46)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)


My working assumption is that I am missing a dependency or import and this is some kind of scala-ism.



I have yet to find any other references to this issue. Ultimately i think it is probably down to imports/dependencies but so far I can't quite see what it is. Any help greatly appreciated. I'm keen to know ways to fix the issue or alternatively to create data frames via less magical approaches than toDf().



Spark info:



Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _ / _ / _ `/ __/ '_/
 /___/ .__/_,_/_/ /_/_ version 2.3.2.0-mapr-1901
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)





scala apache-spark reflection hbase livy







share|improve this question
















I am attempting to produce a Spark Dataframe from within Spark, which has been initialised using apache Livy.



I first noticed this issue on this more complicated hbase call:



 import spark.implicits._

 ... 

 spark.sparkContext
 .newAPIHadoopRDD(
 conf,
 classOf[TableInputFormat],
 classOf[ImmutableBytesWritable],
 classOf[Result]
 )
 .toDF()


But i found i could get the same thing to occur on a simple:



 import spark.implicits._

 ...

 val filtersDf = filters.toDF()


Where, filtersDf is just a sequence of case classes.



The common issue is the *.toDF(), however it also occurs with *.toDS(), which makes me think that the implicit resolution on import spark.implicits._ is not working. The underlying objects to be converted to dataframes do have data.



The error stack looks like it relates to runtime implicit resolution using scala runtime reflection.



Note that i have checked and both spark and the compiled code both use the same version of Scala (2.11) .



The exception i get is:



java.lang.RuntimeException: java.util.NoSuchElementException: head of empty list
scala.collection.immutable.Nil$.head(List.scala:420)
scala.collection.immutable.Nil$.head(List.scala:417)
scala.collection.immutable.List.map(List.scala:277)
scala.reflect.internal.Symbols$Symbol.parentSymbols(Symbols.scala:2117)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:301)
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
scala.reflect.internal.Types$Type.member(Types.scala:600)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
scala.reflect.internal.Mirrors$RootsBase.staticPackage(Mirrors.scala:204)
scala.reflect.runtime.JavaMirrors$JavaMirror.staticPackage(JavaMirrors.scala:82)
scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:263)
scala.reflect.runtime.JavaMirrors$class.scala$reflect$runtime$JavaMirrors$$createMirror(JavaMirrors.scala:32)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:49)
scala.reflect.runtime.JavaMirrors$$anonfun$runtimeMirror$1.apply(JavaMirrors.scala:47)
scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
scala.reflect.runtime.JavaMirrors$class.runtimeMirror(JavaMirrors.scala:46)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)
scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:16)


My working assumption is that I am missing a dependency or import and this is some kind of scala-ism.



I have yet to find any other references to this issue. Ultimately i think it is probably down to imports/dependencies but so far I can't quite see what it is. Any help greatly appreciated. I'm keen to know ways to fix the issue or alternatively to create data frames via less magical approaches than toDf().



Spark info:



Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _ / _ / _ `/ __/ '_/
 /___/ .__/_,_/_/ /_/_ version 2.3.2.0-mapr-1901
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)





scala apache-spark reflection hbase livy




scala apache-spark reflection hbase livy






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 25 at 8:58







ZenMasterZed

















asked Mar 24 at 23:33









ZenMasterZedZenMasterZed

736




736












  • Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

    – Apurba Pandey
    Mar 25 at 5:46











  • I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

    – ZenMasterZed
    Mar 25 at 7:12

















  • Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

    – Apurba Pandey
    Mar 25 at 5:46











  • I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

    – ZenMasterZed
    Mar 25 at 7:12
















Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

– Apurba Pandey
Mar 25 at 5:46





Can you check the count of the rdd before toDF() and check the results? Like, spark.sparkContext.newAPIHadoopRDD(conf,classOf[TableInputFormat],classOf[ImmutableBytesWritable],classOf[Result]).count()

– Apurba Pandey
Mar 25 at 5:46













I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

– ZenMasterZed
Mar 25 at 7:12





I have checked and the rdd has data. But i don't think it is the RDD itself but more related to the toDF call which comes via an implicit. If you look closely at the error it is a scala reflection error.

– ZenMasterZed
Mar 25 at 7:12












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55329603%2fnosuchelementexception-while-using-todf-from-spark-livy%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55329603%2fnosuchelementexception-while-using-todf-from-spark-livy%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

SQL error code 1064 with creating Laravel foreign keysForeign key constraints: When to use ON UPDATE and ON DELETEDropping column with foreign key Laravel error: General error: 1025 Error on renameLaravel SQL Can't create tableLaravel Migration foreign key errorLaravel php artisan migrate:refresh giving a syntax errorSQLSTATE[42S01]: Base table or view already exists or Base table or view already exists: 1050 Tableerror in migrating laravel file to xampp serverSyntax error or access violation: 1064:syntax to use near 'unsigned not null, modelName varchar(191) not null, title varchar(191) not nLaravel cannot create new table field in mysqlLaravel 5.7:Last migration creates table but is not registered in the migration table

Image
Does a multiclassed wizard start with a spellbook? Minimizing medical costs with HSA Boss has banned cycling to work because he thinks it's unsafe Is there a typical layout to blocking installed for backing in new construction framing? In the Seventh Seal why does Death let the chess game happen? Why has Novell never written its own boot loader? Should I warn my boss I might take sick leave Why is the saxophone not common in classical repertoire? Is it possible to spoof an IP address to an exact number? Motorcyle Chain needs to be cleaned every time you lube it? Bypass with wrong cvv of debit card and getting OTP Do we have a much compact and generalized version of erase–remove idiom? how can i make this execution plan more efficient? Why did moving the mouse cursor cause Windows 95 to run more quickly? What's the big deal about the Nazgûl losing their horses? Has chattel slavery ever been used as a criminal punishment in the USA since the passag...
Read more

용인 삼성생명 블루밍스 목차 통계 역대 감독 선수단 응원단 경기장 같이 보기 외부 링크 둘러보기 메뉴samsungblueminx.comeh선수 명단용인 삼성생명 블루밍스용인 삼성생명 블루밍스ehsamsungblueminx.comeheheheh

Image
수원 삼성생명 비추미광주 신세계 쿨캣수원 삼성생명 비추미안산 신한은행 에스버드안산 신한은행 에스버드안산 신한은행 에스버드춘천 우리은행 한새춘천 우리은행 한새춘천 우리은행 한새춘천 우리은행 한새아산 우리은행 위비아산 우리은행 위비청주 KB 스타즈 한국 여자 농구 연맹 팀1977년 설립...
Read more

155 수학 과학 기타 둘러보기 메뉴eh추가해eh문서를 완성해

Image
0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369...
Read more