Investigate throughput for multiple tables in state store committer lambda

[patchwork01]

Background

Related to:

• Support asynchronous commit of all state store updates #2934
• System test state store throughput #2902
• Contention system test with different operations at once #2546

When we added system tests for throughput of the state store committer in StateStoreCommitterThroughputST, we found that it's considerably reduced when we have 10 Sleeper tables.

Description

We'd like to find out why throughput is reduced so much for multiple tables, and see how we can minimise this.

Analysis

We can find more details of the behaviour of throughput on the lambda, since right now we only get the raw throughput per table. We can also look at possible causes.

There are a number of possible causes for this:

• Throughput is limited for the FIFO queue
• Throughput is limited for the Lambda event source
• Multiple tables are processed by a single lambda invocation too often
• Lambda instances spend time coming up to date as tables are swapped between them
• Lambda instances spend time waiting for a retry adding a transaction as tables are swapped between them

Throughput limited

We could enable high throughput for SQS FIFO:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html

We've split out this issue:

• High throughput FIFO queue for state store committer #3134

Multiple tables processed by a single lambda invocation

We could log out statistics for how many tables were processed by a lambda invocation, since we can already get that information from QueryStateStoreCommitterLogs and StateStoreCommitterRuns. We could also assert on that in StateStoreCommitterThroughputST.

We've split out this issue:

• Track multiple Sleeper tables in a single invocation of state store committer #3135

Spending extra time when tables are swapped

We already log out how much time is spent coming up to date in TransactionLogHead. We could add processing of that in QueryStateStoreCommitterLogs and ReadStateStoreCommitterLogs. We could also log more information about the time spent waiting.

We've split out this issue:

• Track Sleeper tables swapping between state store committer Lambda instances #3136

This is also related:

• Chart throughput over time in state store throughput test #3098