How does the kafka consumer of the same group share messages between them?Data Modeling with Kafka? Topics and PartitionsHow Kafka broadcast to many Consumer GroupsKafka Consumer Group Partition issueKafka: Is it possible to share data among consumers in a consumer group?Understanding Kafka Topics and PartitionsKafka Consumer Attached to partition but not consuming messagesJava Kafka consumer group failing to consume a few messagesread filter messages from Kafka topic according to consumerKafka consumer groups behavioral unequallyDoes consumer consume from replica partitions if multiple consumers running under same consumer group?
Can the Produce Flame cantrip be used to grapple, or as an unarmed strike, in the right circumstances?
How did the USSR manage to innovate in an environment characterized by government censorship and high bureaucracy?
What is it called when one voice type sings a 'solo'?
Unbreakable Formation vs. Cry of the Carnarium
Re-submission of rejected manuscript without informing co-authors
What is GPS' 19 year rollover and does it present a cybersecurity issue?
Need help identifying/translating a plaque in Tangier, Morocco
Crop image to path created in TikZ?
Creating a loop after a break using Markov Chain in Tikz
Why is the design of haulage companies so “special”?
Is there a familial term for apples and pears?
Is Social Media Science Fiction?
"listening to me about as much as you're listening to this pole here"
Is there a way to make member function NOT callable from constructor?
LWC and complex parameters
Can I find out the caloric content of bread by dehydrating it?
Is domain driven design an anti-SQL pattern?
What do you call something that goes against the spirit of the law, but is legal when interpreting the law to the letter?
Calculate Levenshtein distance between two strings in Python
Why is my log file so massive? 22gb. I am running log backups
Is ipsum/ipsa/ipse a third person pronoun, or can it serve other functions?
Lied on resume at previous job
Where to refill my bottle in India?
Why Is Death Allowed In the Matrix?
How does the kafka consumer of the same group share messages between them?
Data Modeling with Kafka? Topics and PartitionsHow Kafka broadcast to many Consumer GroupsKafka Consumer Group Partition issueKafka: Is it possible to share data among consumers in a consumer group?Understanding Kafka Topics and PartitionsKafka Consumer Attached to partition but not consuming messagesJava Kafka consumer group failing to consume a few messagesread filter messages from Kafka topic according to consumerKafka consumer groups behavioral unequallyDoes consumer consume from replica partitions if multiple consumers running under same consumer group?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
Say, there is a Consumer group. (Consumers with the same group ID).
The Consumer group is consuming Topic A from a Broker.
Topic A has 4 partitions, and there are 4 Consumers in that group.
Each Consumer consumes different partition. ( Consumer 1 takes messages in partition 1, Consumer 2 takes messages in partition 2 and so on because that's what consumer group does in kafka. Among Consumer Group, each has 1/4 of the topic.
My question : How do they share the message so that they all have Topic A?
How do they combine those bits and pieces? and where does this take place?
If my computer (consumer 1 of group A) consumes Topic A from a Broker, and my friend's computer (consumer 2 of group A) consumes other pieces of the same Topic, how do we combine the message in Topic A?
apache-kafka kafka-consumer-api producer-consumer
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
add a comment |
Say, there is a Consumer group. (Consumers with the same group ID).
The Consumer group is consuming Topic A from a Broker.
Topic A has 4 partitions, and there are 4 Consumers in that group.
Each Consumer consumes different partition. ( Consumer 1 takes messages in partition 1, Consumer 2 takes messages in partition 2 and so on because that's what consumer group does in kafka. Among Consumer Group, each has 1/4 of the topic.
My question : How do they share the message so that they all have Topic A?
How do they combine those bits and pieces? and where does this take place?
If my computer (consumer 1 of group A) consumes Topic A from a Broker, and my friend's computer (consumer 2 of group A) consumes other pieces of the same Topic, how do we combine the message in Topic A?
apache-kafka kafka-consumer-api producer-consumer
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
add a comment |
Say, there is a Consumer group. (Consumers with the same group ID).
The Consumer group is consuming Topic A from a Broker.
Topic A has 4 partitions, and there are 4 Consumers in that group.
Each Consumer consumes different partition. ( Consumer 1 takes messages in partition 1, Consumer 2 takes messages in partition 2 and so on because that's what consumer group does in kafka. Among Consumer Group, each has 1/4 of the topic.
My question : How do they share the message so that they all have Topic A?
How do they combine those bits and pieces? and where does this take place?
If my computer (consumer 1 of group A) consumes Topic A from a Broker, and my friend's computer (consumer 2 of group A) consumes other pieces of the same Topic, how do we combine the message in Topic A?
apache-kafka kafka-consumer-api producer-consumer
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
Say, there is a Consumer group. (Consumers with the same group ID).
The Consumer group is consuming Topic A from a Broker.
Topic A has 4 partitions, and there are 4 Consumers in that group.
Each Consumer consumes different partition. ( Consumer 1 takes messages in partition 1, Consumer 2 takes messages in partition 2 and so on because that's what consumer group does in kafka. Among Consumer Group, each has 1/4 of the topic.
My question : How do they share the message so that they all have Topic A?
How do they combine those bits and pieces? and where does this take place?
If my computer (consumer 1 of group A) consumes Topic A from a Broker, and my friend's computer (consumer 2 of group A) consumes other pieces of the same Topic, how do we combine the message in Topic A?
apache-kafka kafka-consumer-api producer-consumer
apache-kafka kafka-consumer-api producer-consumer
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
edited Mar 22 at 5:09
Jin Lee
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
asked Mar 22 at 1:55
Jin LeeJin Lee
127215
127215
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I thought of the term 'Consumer' a computer or a server consuming a Topic from a Broker. That's why I got confused with Consumer group.
Consumer is a client or a program, and I can have many consumer's on my computer or server. Consumer Group means multiple consumer processes on an independent machine
So I don't need to worry about Consumer's in a group sharing bits of message to complete a Topic. Previously, I thought each consumer being a server or a computing resource, so they had to communicate somehow. But that's how I got confused. They don't need to communicate to each other over the network or need a pool to share their consumed partitioned.
Consumer 1 can read from partition 1, Consumer 2 can read from partition 2, and if Consumer 1, 2 share the same group ID (Consumer group), Consumer 1 doesn't need to read from partition 2, and Consumer 2 doesn't need to read from partion 1. They already have a Topic they need. Boom!
I posted an answer to help someone who thought like me.
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55291786%2fhow-does-the-kafka-consumer-of-the-same-group-share-messages-between-them%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I thought of the term 'Consumer' a computer or a server consuming a Topic from a Broker. That's why I got confused with Consumer group.
Consumer is a client or a program, and I can have many consumer's on my computer or server. Consumer Group means multiple consumer processes on an independent machine
So I don't need to worry about Consumer's in a group sharing bits of message to complete a Topic. Previously, I thought each consumer being a server or a computing resource, so they had to communicate somehow. But that's how I got confused. They don't need to communicate to each other over the network or need a pool to share their consumed partitioned.
Consumer 1 can read from partition 1, Consumer 2 can read from partition 2, and if Consumer 1, 2 share the same group ID (Consumer group), Consumer 1 doesn't need to read from partition 2, and Consumer 2 doesn't need to read from partion 1. They already have a Topic they need. Boom!
I posted an answer to help someone who thought like me.
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
add a comment |
I thought of the term 'Consumer' a computer or a server consuming a Topic from a Broker. That's why I got confused with Consumer group.
Consumer is a client or a program, and I can have many consumer's on my computer or server. Consumer Group means multiple consumer processes on an independent machine
So I don't need to worry about Consumer's in a group sharing bits of message to complete a Topic. Previously, I thought each consumer being a server or a computing resource, so they had to communicate somehow. But that's how I got confused. They don't need to communicate to each other over the network or need a pool to share their consumed partitioned.
Consumer 1 can read from partition 1, Consumer 2 can read from partition 2, and if Consumer 1, 2 share the same group ID (Consumer group), Consumer 1 doesn't need to read from partition 2, and Consumer 2 doesn't need to read from partion 1. They already have a Topic they need. Boom!
I posted an answer to help someone who thought like me.
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
add a comment |
I thought of the term 'Consumer' a computer or a server consuming a Topic from a Broker. That's why I got confused with Consumer group.
Consumer is a client or a program, and I can have many consumer's on my computer or server. Consumer Group means multiple consumer processes on an independent machine
So I don't need to worry about Consumer's in a group sharing bits of message to complete a Topic. Previously, I thought each consumer being a server or a computing resource, so they had to communicate somehow. But that's how I got confused. They don't need to communicate to each other over the network or need a pool to share their consumed partitioned.
Consumer 1 can read from partition 1, Consumer 2 can read from partition 2, and if Consumer 1, 2 share the same group ID (Consumer group), Consumer 1 doesn't need to read from partition 2, and Consumer 2 doesn't need to read from partion 1. They already have a Topic they need. Boom!
I posted an answer to help someone who thought like me.
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
I thought of the term 'Consumer' a computer or a server consuming a Topic from a Broker. That's why I got confused with Consumer group.
Consumer is a client or a program, and I can have many consumer's on my computer or server. Consumer Group means multiple consumer processes on an independent machine
So I don't need to worry about Consumer's in a group sharing bits of message to complete a Topic. Previously, I thought each consumer being a server or a computing resource, so they had to communicate somehow. But that's how I got confused. They don't need to communicate to each other over the network or need a pool to share their consumed partitioned.
Consumer 1 can read from partition 1, Consumer 2 can read from partition 2, and if Consumer 1, 2 share the same group ID (Consumer group), Consumer 1 doesn't need to read from partition 2, and Consumer 2 doesn't need to read from partion 1. They already have a Topic they need. Boom!
I posted an answer to help someone who thought like me.
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
answered Mar 22 at 7:14
Jin LeeJin Lee
127215
127215
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55291786%2fhow-does-the-kafka-consumer-of-the-same-group-share-messages-between-them%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
