Styjun

I have implemented spark streaming and currently testing this. I have submitted the program and it looks good. But all of a sudden, after 10 hrs, it started behaving weirdly.

The average number of records per batch (10 secs) would be around 600K messages. But all of a sudden the number of messages for few batches is coming to 1000 though there are thousands of messages. And then the number increases to millions as mentioned below. Also, the total delay is huge when the number of messages is in thousand

log - Batch completed, Total delay :19786 ms for messages:1000
log - Batch completed, Total delay :18787 ms for messages:1000
log - Batch completed, Total delay :16476 ms for messages:1000
log - Batch completed, Total delay :13354 ms for messages:1000
log - Batch completed, Total delay :10290 ms for messages:1000
...
...
log - Batch completed, Total delay :5103 ms for messages:311500
log - Batch completed, Total delay :7599 ms for messages:610250
log - Batch completed, Total delay :7358 ms for messages:803000
log - Batch completed, Total delay :7833 ms for messages:1091250
log - Batch completed, Total delay :7219 ms for messages:1393250
log - Batch completed, Total delay :9601 ms for messages:1930000
log - Batch completed, Total delay :8718 ms for messages:2010250

Not sure if these low number of messages are backlog messages or due to some other config. Could you please let me know why all of a sudden am I getting only 1000 messages in a batch and then huge spike in the count. Are these messages (in 1000) from old batches?

Also, am not sure why the delay time is too high for these low messages though the execution time for the batch is less than 10s

Total delay: 113.748 s for time 1553181660000 ms (execution: 8.314 s)

Is this because they are from old batches?

Thanks