Styjun

Why does the U.S. tolerate foreign influence from Saudi Arabia and Israel on its domestic policies while not tolerating that from China or Russia?

SOLVED - GFCI - should my neutral and ground have continuity?

As the Dungeon Master, how do I handle a player that insists on a specific class when I already know that choice will cause issues?

Domain of definition of Laplace Operator on L^2

Flatten array with OPENJSON: OPENJSON on a value that may not be an array? [ [1] ], vs [1]

Is anyone advocating the promotion of homosexuality in UK schools?

How might the United Kingdom become a republic?

<schwitz>, <zwinker> etc. Does German always use 2nd Person Singular Imperative verbs for emoticons? If so, why?

What's the minimum number of sensors for a hobby GPS waypoint-following UAV?

Why isn't there research to build a standard lunar, or Martian mobility platform?

Supporting developers who insist on using their pet language

Can I intentionally omit previous work experience or pretend it doesn't exist when applying for jobs?

Can I play a first turn Simic Growth Chamber to have 3 mana available in the second turn?

What do the horizontal lines in a P-V phase diagram mean?

Why do players in the past play much longer tournaments than today's top players?

Simple LED driver, transistor and GPIO

Get ids only where one id is null and other isn't

Referring to different instances of the same character in time travel

PIC12F675 GP4 doesn't work

Why are they 'nude photos'?

What does "it kind of works out" mean?

Robbers: The Hidden OEIS Substring

Why were Er and Onan punished if they were under 20?

Is an acid a salt or not?

Spark Structured Streaming, Writing into HDFS & Running Alter table on Hive dynamically

How to write Spark Structured Streaming Data into Hive?How to write streaming dataset into Hive?Spark Structured Streaming - Insert into existing Hive table with scalability without errorshow to read orc transaction hive table in spark?spark streaming persistent table updatesSpark Structured Streaming Writestream to Hive ORC Partioned External TableHow to insert spark structured streaming DataFrame to Hive external table/location?spark structured streaming producing .c000.csv filesHow to write data to hive table with snappy compression in Spark SQLspark structured stream read to hdfs files fails if data is read immediately

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

My use case is to
1. Read data from Kafka, parse it and write into HDFS as ORC files.
2. Run a alter table query on Hive table dynamically on date column from data.

I am able to do the step 1 easily(writing into HDFS)
But for step 2(alter table query), I have tried foreachwriter. But As my cluster is keborized, establishing JDBC is hard. I have tried setting up all the configs, but still facing issues with foreachwriter.

Also I want to make sure Step 2 should after step 1 for all the microbatches. I am not sure how spark structured streaming behaves in this scenario?

Is there any other way to achieve step 2?

Thanks in advance.

apache-spark spark-streaming spark-structured-streaming

share|improve this question

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

add a comment | 

0

My use case is to
1. Read data from Kafka, parse it and write into HDFS as ORC files.
2. Run a alter table query on Hive table dynamically on date column from data.

I am able to do the step 1 easily(writing into HDFS)
But for step 2(alter table query), I have tried foreachwriter. But As my cluster is keborized, establishing JDBC is hard. I have tried setting up all the configs, but still facing issues with foreachwriter.

Also I want to make sure Step 2 should after step 1 for all the microbatches. I am not sure how spark structured streaming behaves in this scenario?

Is there any other way to achieve step 2?

Thanks in advance.

apache-spark spark-streaming spark-structured-streaming

share|improve this question

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

add a comment | 

My use case is to
1. Read data from Kafka, parse it and write into HDFS as ORC files.
2. Run a alter table query on Hive table dynamically on date column from data.

I am able to do the step 1 easily(writing into HDFS)
But for step 2(alter table query), I have tried foreachwriter. But As my cluster is keborized, establishing JDBC is hard. I have tried setting up all the configs, but still facing issues with foreachwriter.

Also I want to make sure Step 2 should after step 1 for all the microbatches. I am not sure how spark structured streaming behaves in this scenario?

Is there any other way to achieve step 2?

Thanks in advance.

apache-spark spark-streaming spark-structured-streaming

share|improve this question

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

share|improve this question

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

share|improve this question

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges

asked Mar 26 at 3:57

Siva kumarSiva kumar

122 bronze badges