Increase message throughput

[steffenstolze]

Hey there!
We're trying to scale our message processor (Node.js program running on Fargate) to eventually support fetching and processing 1000+ messages per second.
Right now I'm not even able to fetch 400 messages per second. I tried my local MacBook (M1 Pro) as well as different ECS Fargate task configurations (Batch processing, 10 tasks with 1vCPU or 5 tasks with 4 vCPU and 4 node workers, etc) but never reached more than ~350 messages per second (fetch them and increase a counter).
This is the point where I ask myself if Lambda would be the better alternative here, or maybe even another runtime.

What is your experience with high throughput requirements?

Thanks a lot!

[nicholasgriffintn]

Hey! I've asked around and I no one internally is really using SQS Consumer this way, so we don't have much advice on this unfortunately.

Hopefully someone might come along with some advice or you might be able to ask the AWS team on their thoughts? They have some great people on twitter who know a thing or two.

[zinosama]

@nicholasgriffintn we ran into the same issue recently. Similar setup - ECS on Fargate. We saw a backlog of messages in the queue and so we tried increasing task (container) count. However, we noticed the increase in the container count led to a proportional increase in the response time of each aws network call to pull messages. Consequently, the combined throughput remained at the same level as before the scale-up.

Below is a screenshot of processed message count by container that shows how the combined throughput remained the same despite of a larger number of containers running the consumer:

Below is a screenshot demonstrating the increase in message-poll response time:

I'm curious if you have ever came across similar issues.

Also you mentioned

no one internally is really using SQS Consumer this way
I'm pretty new to SQS so curious to learn how you guys tend to run SQS consumer processes.

Many thanks in advance.

[utkarshdeep]

steffenstolze Any breakthrough on this?