Styjun

I have a SQS consumer running in EventConsumerService that needs to handle up to 3K TPS successfully, sometimes upwards of 20K TPS (or 1.2 million messages per minute). For each message processed, I make a REST call to DataService's TCP VIP. I'm trying to perform a load test to find the max TPS that one host can handle in EventConsumerService without overstraining:

1. Request volume on dependencies, DynamoDB storage, etc


2. CPU utilization in both EventConsumerService and DataService


3. Network connections per host


4. IO stats due to overlogging


5. DLQ size must be minimal, currently I am seeing my DLQ growing to 500K messages due to 500 Service Unavailable exceptions thrown from DataService, so something must be wrong.


6. Approximate age of oldest message. I do not want a message sitting in the queue for over X minutes.


7. Fatals and latency of the REST call to DataService


8. Active threads

This is how I am performing the performance test:

1. I set up both my consumer and the other service on one host, the reason being I want to understand the load on both services per host.




2. I use a TPS generator to fill the SQS queue with a million messages




3. The EventConsumerService service is already running in production. Once messages started filling the SQS queue, I immediately could see requests being sent to DataService.

Here are the parameters I am tuning to find messagesPolledPerSecond:

messagesPolledPerSecond = (numberOfHosts * numberOfPollers * messageFetchSize) * (1000/(sleepTimeBetweenPollsPerMs+receiveMessageTimePerMs))
messagesInSurge / messagesPolledPerSecond = ageOfOldestMessageSLA
ageOfOldestMessage + settingsUpdatedLatency < latencySLA

The variables for SqsConsumer which I kept constant are:

1. numberOfHosts = 1


2. ReceiveMessageTimePerMs = 60 ms? It's out of my control


3. Max thread pool size: 300

Other factors are all game:

1. Number of pollers (default 1), I set to 150


2. Sleep time between polls (default 100 ms), I set to 0 ms


3. Sleep time when no messages (default 1000 ms), ???


4. message fetch size (default 1), I set to 10

However, with the above parameters, I am seeing a high amount of messages being sent to the DLQ due to server errors, so clearly I have set values to be too high. This testing methodology seems highly inefficient, and I am unable to find the optimal TPS that does not cause such a tremendous number of messages to be sent to the DLQ, and does not cause such a high approximate age of the oldest message.

Any guidance is appreciated in how best I should test. It'd be very helpful if we can set up a time to chat. PM me directly