Styjun

Can you help me, to widen the page. Thank you

How to extract coefficients of a generating function like this one, using a computer?

What's the idiomatic (or best) way to trim surrounding whitespace from a string?

What does this Pokemon Trainer mean by saying the player is "SHELLOS"?

How might boat designs change in order to allow them to be pulled by dragons?

Was Wolfgang Unziker the last Amateur GM?

"Best practices" for formulating MIPs

Can I hire several veteran soldiers to accompany me?

Emphasize numbers in tables

Odd PCB Layout for Voltage Regulator

How to model a Coral or Sponge Structure?

Other homotopy invariants?

Are there advantages in writing by hand over typing out a story?

How can I write this expression using summation or double summation?

Classify 2-dim p-adic galois representations

Using PTSerif-TLF for Cyrillic with TeX Gyre Pagella

Why are symbols not written in words?

Why would a propellor have blades of different lengths?

usage of y" not just for locations?

Cannot overlay, because ListPlot does not draw same X range despite the same PlotRange

Russian equivalents of 能骗就骗 (if you can cheat, then cheat)

Why will we fail creating a self sustaining off world colony?

How can I change my buffer system for protein purification?

German idiomatic equivalents of 能骗就骗 (if you can cheat, then cheat)

Get last committed message in a partition kafka

Apache Kafka Topic Partition Message HandlingWhat ways can a Consumer consume message in Kafka?Understanding Kafka Topics and PartitionsProblems with the retention period for offset topic of kafkakafka consumer fetch API does not return the right offset valuehow to get last committed offset from read_committed Kafka ConsumerKafka Transactional Producer & ConsumerKafka, consumer offset and multiple async commitsKafka: Polling after producer transaction doesn't get the produced messagesKafka last message poll results in 0 messages

3

I am using exactly once semantics provided by kafka. Therefore, my producer writes a message within a transaction. While my producer was sending 100th message, right between send() and commitTransaction(); I killed the producer process.

I read last few uncommitted messages in my topic.

Consumer record offset - message number

0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed

Now, when I run consumer with read_committed isolation level. It exactly reads from 1-99 messages. But for that I have read entire topic. Eventually, I am going to store millions of messages in a topic. So, reading entire topic is not preferable.

Also, assume that consumer is polling messages from a broker and there is some communication issue with a kafka borker and consumer. Last message read by consumer is let's say offset#50. This means that I could not identify the last committed message in a topic reliably.

I used other methods i.e.

seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200

Is there a way to get message that was committed by the Kafka producer reliably ? (In my case, Offset#196)

apache-kafka kafka-consumer-api

share|improve this question

edited Mar 25 at 20:19

JR ibkr

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you share the code of your consumer? Also, are you sure that the last committed offset is 198?
  
  – Giorgos Myrianthous
  Mar 25 at 18:03
  
  

  
  
  
  

add a comment | 

3

I am using exactly once semantics provided by kafka. Therefore, my producer writes a message within a transaction. While my producer was sending 100th message, right between send() and commitTransaction(); I killed the producer process.

I read last few uncommitted messages in my topic.

Consumer record offset - message number

0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed

Now, when I run consumer with read_committed isolation level. It exactly reads from 1-99 messages. But for that I have read entire topic. Eventually, I am going to store millions of messages in a topic. So, reading entire topic is not preferable.

Also, assume that consumer is polling messages from a broker and there is some communication issue with a kafka borker and consumer. Last message read by consumer is let's say offset#50. This means that I could not identify the last committed message in a topic reliably.

I used other methods i.e.

seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200

Is there a way to get message that was committed by the Kafka producer reliably ? (In my case, Offset#196)

apache-kafka kafka-consumer-api

share|improve this question

edited Mar 25 at 20:19

JR ibkr

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you share the code of your consumer? Also, are you sure that the last committed offset is 198?
  
  – Giorgos Myrianthous
  Mar 25 at 18:03
  
  

  
  
  
  

add a comment | 

I am using exactly once semantics provided by kafka. Therefore, my producer writes a message within a transaction. While my producer was sending 100th message, right between send() and commitTransaction(); I killed the producer process.

I read last few uncommitted messages in my topic.

Consumer record offset - message number

0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed

Now, when I run consumer with read_committed isolation level. It exactly reads from 1-99 messages. But for that I have read entire topic. Eventually, I am going to store millions of messages in a topic. So, reading entire topic is not preferable.

Also, assume that consumer is polling messages from a broker and there is some communication issue with a kafka borker and consumer. Last message read by consumer is let's say offset#50. This means that I could not identify the last committed message in a topic reliably.

I used other methods i.e.

seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200

Is there a way to get message that was committed by the Kafka producer reliably ? (In my case, Offset#196)

apache-kafka kafka-consumer-api

share|improve this question

edited Mar 25 at 20:19

JR ibkr

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

Consumer record offset - message number

0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed

seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200

share|improve this question

edited Mar 25 at 20:19

JR ibkr

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

share|improve this question

edited Mar 25 at 20:19

JR ibkr

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges

asked Mar 25 at 16:56

JR ibkrJR ibkr

41911 silver badge1313 bronze badges