Spring Cloud Aws kinesis Binder All JVMs in group consume same message at different intervalsAll the consumers from consumer group receive messagespring-cloud-stream-kafka not honoring single consumer for a groupCan I bind to multiple consumer groups with Spring Cloud Stream?Implement Custom error message handler in Spring Cloud Aws Kinesis StreamSpring cloud aws stream, messages are consumed by multiple instances in consumer groupSpring Integration Kinesis adapter and consumer groupsspring cloud stream kinesis BinderSpring Cloud Stream Consumer Load Balancing StrategyCannot retrieve binder configuration in spring-cloud-stream 2.1.0Spring Cloud Stream Kinesis Group in autoscaling group over JdbcLockRegistry
Is this story about US tax office reasonable?
Smart people send dumb people to a new planet on a space craft that crashes into a body of water
Do firearms count as ranged weapons?
I think I may have violated academic integrity last year - what should I do?
How to properly maintain eye contact with people that have distinct facial features?
Why are huge challot displayed on the wedding couple / Bar Mitzvah's table?
Glitch in AC sine wave interfering with phase cut dimming
How do you deal with an abrupt change in personality for a protagonist?
Black-and-white film where monster/alien gets fried
Preserving culinary oils
What's the connection between "kicking a pigeon" and "how a bill becomes a law"?
Why colon to denote that a value belongs to a type?
What does "Marchentalender" on the front of a postcard mean?
What is a subpixel in Super Mario Bros, and how does it relate to wall clipping?
Plot exactly N bounce of a ball
What are the benefits of cryosleep?
How were these pictures of spacecraft wind tunnel testing taken?
Is a post-climate apocolypse city in which many or most insects have disappeared realistic?
What F1 in name of seeds/varieties means?
How feasible is the Delta-Glider?
How to extract lower and upper bound in numeric format from a confidence interval string?
Comment dit-on « I’ll tell you what » ?
Why does the UK have more political parties than the US?
Apparent Ring of Craters on the Moon
Spring Cloud Aws kinesis Binder All JVMs in group consume same message at different intervals
All the consumers from consumer group receive messagespring-cloud-stream-kafka not honoring single consumer for a groupCan I bind to multiple consumer groups with Spring Cloud Stream?Implement Custom error message handler in Spring Cloud Aws Kinesis StreamSpring cloud aws stream, messages are consumed by multiple instances in consumer groupSpring Integration Kinesis adapter and consumer groupsspring cloud stream kinesis BinderSpring Cloud Stream Consumer Load Balancing StrategyCannot retrieve binder configuration in spring-cloud-stream 2.1.0Spring Cloud Stream Kinesis Group in autoscaling group over JdbcLockRegistry
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
We have three containers connecting to Kinesis stream as consumers using Spring cloud AWS kinesis binder.
We have used consumer groups for load balancing across three containers. The requirement is that containers share the load and it shall be possible to distribute the load evenly.
- All containers use the same application config from config server
Currently we have configuration like below.
spring:
cloud:
stream:
bindings:
MyStream:
group: my-group
destination: stream-1
content-type: application/json
All containers are consuming same message, but at the different times (having difference of 5 to 10 mins)
As per the documentation,
Static shard distribution within a single consumer group
It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring:
spring.cloud.stream.instanceCount= to number of instances
spring.cloud.stream.instanceIndex= current instance’s index
As we use the same config server, can you please help how to make sure that load balancing is achieved.
spring-cloud-stream spring-cloud-aws spring-integration-aws
asked Mar 24 at 8:43
PatanPatan
5,3352786149
add a comment |
We have three containers connecting to Kinesis stream as consumers using Spring cloud AWS kinesis binder.
We have used consumer groups for load balancing across three containers. The requirement is that containers share the load and it shall be possible to distribute the load evenly.
- All containers use the same application config from config server
Currently we have configuration like below.
spring:
cloud:
stream:
bindings:
MyStream:
group: my-group
destination: stream-1
content-type: application/json
All containers are consuming same message, but at the different times (having difference of 5 to 10 mins)
As per the documentation,
Static shard distribution within a single consumer group
It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring:
spring.cloud.stream.instanceCount= to number of instances
spring.cloud.stream.instanceIndex= current instance’s index
As we use the same config server, can you please help how to make sure that load balancing is achieved.
spring-cloud-stream spring-cloud-aws spring-integration-aws
asked Mar 24 at 8:43
PatanPatan
5,3352786149
add a comment |
We have three containers connecting to Kinesis stream as consumers using Spring cloud AWS kinesis binder.
We have used consumer groups for load balancing across three containers. The requirement is that containers share the load and it shall be possible to distribute the load evenly.
- All containers use the same application config from config server
Currently we have configuration like below.
spring:
cloud:
stream:
bindings:
MyStream:
group: my-group
destination: stream-1
content-type: application/json
All containers are consuming same message, but at the different times (having difference of 5 to 10 mins)
As per the documentation,
Static shard distribution within a single consumer group
It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring:
spring.cloud.stream.instanceCount= to number of instances
spring.cloud.stream.instanceIndex= current instance’s index
As we use the same config server, can you please help how to make sure that load balancing is achieved.
spring-cloud-stream spring-cloud-aws spring-integration-aws
asked Mar 24 at 8:43
PatanPatan
5,3352786149
We have three containers connecting to Kinesis stream as consumers using Spring cloud AWS kinesis binder.
We have used consumer groups for load balancing across three containers. The requirement is that containers share the load and it shall be possible to distribute the load evenly.
- All containers use the same application config from config server
Currently we have configuration like below.
spring:
cloud:
stream:
bindings:
MyStream:
group: my-group
destination: stream-1
content-type: application/json
All containers are consuming same message, but at the different times (having difference of 5 to 10 mins)
As per the documentation,
Static shard distribution within a single consumer group
It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring:
spring.cloud.stream.instanceCount= to number of instances
spring.cloud.stream.instanceIndex= current instance’s index
As we use the same config server, can you please help how to make sure that load balancing is achieved.
spring-cloud-stream spring-cloud-aws spring-integration-aws
spring-cloud-stream spring-cloud-aws spring-integration-aws
asked Mar 24 at 8:43
PatanPatan
5,3352786149
asked Mar 24 at 8:43
PatanPatan
5,3352786149
asked Mar 24 at 8:43
PatanPatan
5,3352786149
asked Mar 24 at 8:43
PatanPatan
5,3352786149
asked Mar 24 at 8:43
PatanPatan
5,3352786149
5,3352786149
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.
It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.
For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55322020%2fspring-cloud-aws-kinesis-binder-all-jvms-in-group-consume-same-message-at-differ%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.
It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.
For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
add a comment |
First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.
It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.
For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
add a comment |
First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.
It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.
For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.
It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.
For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
answered Mar 25 at 14:04
Artem BilanArtem Bilan
69.7k85074
69.7k85074
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
add a comment |
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
Thank you Artem. I guess we did not had the StreamListener in our application and messages were coming to Transformer directly as Trasnformer(input), may be because of this, the issue had come. After putting the StreamListener, everything works good.
– Patan
Mar 27 at 18:04
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55322020%2fspring-cloud-aws-kinesis-binder-all-jvms-in-group-consume-same-message-at-differ%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
