Streaming job vs looped batch job for consuming data from Kafka queueWhy no Kafka data ingestion for some batches?Spark Streaming job behaves like a batchMake Spark Streaming start Spark Batch jobsSpark Batch is pausing Spark Streaming jobRunning kafka consumer(new Consumer API) foreverHow to reprocess a batched Kafka StreamConsume kafka data to HDFS Spark BatchBest practices for Kafka streamsSpark Streaming Kafka Stream batch executionManaging Offsets with Spark Structured Batch Job with Kafka
Videos of surgery
What was the point of "Substance"?
Notice period 60 days but I need to join in 45 days
Is there a word or phrase that means "use other people's wifi or Internet service without consent"?
74S vs 74LS ICs
Which meaning of "must" does the Slow spell use?
Why is there not a willingness from the world to step in between Pakistan and India?
Can a character use multiple reactions in response to the same trigger?
Unlock your Lock
Half filled water bottle
Why did James Cameron decide to give Alita big eyes?
How do I insert two edge loops equally spaced from the edges?
Did the Apollo Guidance Computer really use 60% of the world's ICs in 1963?
Alternatives to Network Backup
Why did the population of Bhutan drop by 70% between 2007 and 2008?
Stolen MacBook should I worry about my data?
A first "Hangman" game in Python
If I said I had $100 when asked, but I actually had $200, would I be lying by omission?
Grep contents before a colon
助けてくれて有難う meaning and usage
Why does matter stay collapsed in the core, following a supernova explosion?
Many many thanks
Why does a sticker slowly peel off, but if it is pulled quickly it tears?
Can an object tethered to a spaceship be pulled out of event horizon?
Streaming job vs looped batch job for consuming data from Kafka queue
Why no Kafka data ingestion for some batches?Spark Streaming job behaves like a batchMake Spark Streaming start Spark Batch jobsSpark Batch is pausing Spark Streaming jobRunning kafka consumer(new Consumer API) foreverHow to reprocess a batched Kafka StreamConsume kafka data to HDFS Spark BatchBest practices for Kafka streamsSpark Streaming Kafka Stream batch executionManaging Offsets with Spark Structured Batch Job with Kafka
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
For example, if I have a Java application that is always running and it's reading data from a Kafka queue as it arrives, with the purpose of getting data and forwarding it to a db... Is there any major downside to infinitely looping through a batch-job rather than doing it through Kafka Streaming/Kafka Connect? Considering I'm not pointlessly creating or wasting resources each loop and that I'm handling the multithreading properly, are there any major downsides or reasons not to do this? Is it a viable option for a long running application?
(I have reasons for not using Kafka Streams/Connect with our use case)
java loops apache-kafka streaming kafka-consumer-api
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
add a comment |
For example, if I have a Java application that is always running and it's reading data from a Kafka queue as it arrives, with the purpose of getting data and forwarding it to a db... Is there any major downside to infinitely looping through a batch-job rather than doing it through Kafka Streaming/Kafka Connect? Considering I'm not pointlessly creating or wasting resources each loop and that I'm handling the multithreading properly, are there any major downsides or reasons not to do this? Is it a viable option for a long running application?
(I have reasons for not using Kafka Streams/Connect with our use case)
java loops apache-kafka streaming kafka-consumer-api
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
add a comment |
For example, if I have a Java application that is always running and it's reading data from a Kafka queue as it arrives, with the purpose of getting data and forwarding it to a db... Is there any major downside to infinitely looping through a batch-job rather than doing it through Kafka Streaming/Kafka Connect? Considering I'm not pointlessly creating or wasting resources each loop and that I'm handling the multithreading properly, are there any major downsides or reasons not to do this? Is it a viable option for a long running application?
(I have reasons for not using Kafka Streams/Connect with our use case)
java loops apache-kafka streaming kafka-consumer-api
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
For example, if I have a Java application that is always running and it's reading data from a Kafka queue as it arrives, with the purpose of getting data and forwarding it to a db... Is there any major downside to infinitely looping through a batch-job rather than doing it through Kafka Streaming/Kafka Connect? Considering I'm not pointlessly creating or wasting resources each loop and that I'm handling the multithreading properly, are there any major downsides or reasons not to do this? Is it a viable option for a long running application?
(I have reasons for not using Kafka Streams/Connect with our use case)
java loops apache-kafka streaming kafka-consumer-api
java loops apache-kafka streaming kafka-consumer-api
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
edited Apr 3 at 22:23
Artanis
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
asked Mar 27 at 20:03
ArtanisArtanis
32517 bronze badges
32517 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
So I talked to my staff engineer. It turns out that it is a viable option to do an infinite loop through a batch job as long as you're doing it correctly. Meaning you've fine tuned the configuration, your pausing briefly between loops, and you're properly handling the life of any connections/consumers/producers if they were to fail. Another team is already doing it in prod environment.
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55385576%2fstreaming-job-vs-looped-batch-job-for-consuming-data-from-kafka-queue%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
So I talked to my staff engineer. It turns out that it is a viable option to do an infinite loop through a batch job as long as you're doing it correctly. Meaning you've fine tuned the configuration, your pausing briefly between loops, and you're properly handling the life of any connections/consumers/producers if they were to fail. Another team is already doing it in prod environment.
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
add a comment |
So I talked to my staff engineer. It turns out that it is a viable option to do an infinite loop through a batch job as long as you're doing it correctly. Meaning you've fine tuned the configuration, your pausing briefly between loops, and you're properly handling the life of any connections/consumers/producers if they were to fail. Another team is already doing it in prod environment.
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
add a comment |
So I talked to my staff engineer. It turns out that it is a viable option to do an infinite loop through a batch job as long as you're doing it correctly. Meaning you've fine tuned the configuration, your pausing briefly between loops, and you're properly handling the life of any connections/consumers/producers if they were to fail. Another team is already doing it in prod environment.
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
So I talked to my staff engineer. It turns out that it is a viable option to do an infinite loop through a batch job as long as you're doing it correctly. Meaning you've fine tuned the configuration, your pausing briefly between loops, and you're properly handling the life of any connections/consumers/producers if they were to fail. Another team is already doing it in prod environment.
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
answered Apr 3 at 22:11
ArtanisArtanis
32517 bronze badges
32517 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55385576%2fstreaming-job-vs-looped-batch-job-for-consuming-data-from-kafka-queue%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
