state setTimeoutDuration not working in spark structured streamingWhy does Spark application fail with “ClassNotFoundException: Failed to find data source: kafka” as uber-jar with sbt assembly?How to use from_json with Kafka connect 0.10 and Spark Structured Streaming?Spark structured streaming- python - org.apache.kafka.common.TopicPartition; class invalid for deserializationHow to pivot streaming dataset?Structured streaming won't write DF to file sink citing /_spark_metadata/9.compact doesn't existSpark can't identify the event time column being supplied to withWatermark()Convert a Spark SQL batch source to structured streaming sinkOld Kafka Offset consuming by Spark Structured Streaming after clearing Checkpointing locationJava code for reading Kafka avro messages in spark 2.1.1 structure streamingSpark 2.4.0 Structured Streaming Set Kafka consumer group.id
Is mountain bike good for long distances?
What makes things real?
Why does 8 bit truecolor use only 2 bits for blue?
The meaning of "offing" in "an agreement in the offing"
Why did Tony's Arc Reactor do this?
How to run NPCs with complicated mechanics?
How should Thaumaturgy's "three times as loud as normal" be interpreted?
More than three domains hosted on the same IP address
Did the Byzantines ever attempt to move their capital to Rome?
PWM on 5V GPIO pin
Why are some hotels asking you to book through Booking.com instead of matching the price at the front desk?
Strategies for dealing with chess burnout?
How can faith be maintained in a world of living gods?
Capacitors with same voltage, same capacitance, same temp, different diameter?
Supervisor wants me to support a diploma-thesis SW tool after I graduated
Why are UK MPs allowed to abstain (but it counts as a no)?
is it possible to change a material depending on whether it is intersecting with another object?
When Conservative MPs lose the whip, do they retain the right to vote in leadership elections?
How can I hint that my character isn't real?
Word for something that used to be popular but not anymore
Leaving the USA for 10 yrs when you have asylum
Why is it that I have to play this note on the piano as A sharp?
I need to know information from an old German birth certificate
Project Euler problem #112
state setTimeoutDuration not working in spark structured streaming
Why does Spark application fail with “ClassNotFoundException: Failed to find data source: kafka” as uber-jar with sbt assembly?How to use from_json with Kafka connect 0.10 and Spark Structured Streaming?Spark structured streaming- python - org.apache.kafka.common.TopicPartition; class invalid for deserializationHow to pivot streaming dataset?Structured streaming won't write DF to file sink citing /_spark_metadata/9.compact doesn't existSpark can't identify the event time column being supplied to withWatermark()Convert a Spark SQL batch source to structured streaming sinkOld Kafka Offset consuming by Spark Structured Streaming after clearing Checkpointing locationJava code for reading Kafka avro messages in spark 2.1.1 structure streamingSpark 2.4.0 Structured Streaming Set Kafka consumer group.id
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am using the example of spark structured streaming "https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala"
version 2.3.0; after specified timeout duration the state is not expired but the same code working as expected in version 2.4.0. are you aware of what fix has been applied so that we can apply the PR in 2.3.0 spark version
spark-structured-streaming
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
add a comment |
I am using the example of spark structured streaming "https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala"
version 2.3.0; after specified timeout duration the state is not expired but the same code working as expected in version 2.4.0. are you aware of what fix has been applied so that we can apply the PR in 2.3.0 spark version
spark-structured-streaming
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55
add a comment |
I am using the example of spark structured streaming "https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala"
version 2.3.0; after specified timeout duration the state is not expired but the same code working as expected in version 2.4.0. are you aware of what fix has been applied so that we can apply the PR in 2.3.0 spark version
spark-structured-streaming
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
I am using the example of spark structured streaming "https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala"
version 2.3.0; after specified timeout duration the state is not expired but the same code working as expected in version 2.4.0. are you aware of what fix has been applied so that we can apply the PR in 2.3.0 spark version
spark-structured-streaming
spark-structured-streaming
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
asked Mar 28 at 6:50
Selvam RamanSelvam Raman
74 bronze badges
74 bronze badges
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55
add a comment |
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55391676%2fstate-settimeoutduration-not-working-in-spark-structured-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55391676%2fstate-settimeoutduration-not-working-in-spark-structured-streaming%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
In spark version 2.3.0; if i am giving the same words repeated time ever after specified timeout duration it is not expired. But, it does if i am passing different words then the previous state expired; For example line 1 - "a b c"; after 20 seconds; line 2 - "a b c" the state is not expired even after 1 minute; if i am giving different value in line 3 - "s" then the previous state of "a, b, c" keys got expired. This is not the behavior in spark version 2.4.0(could be solved in version between 2.3 to 2.4)
– Selvam Raman
Mar 28 at 6:55