Add asynchronous ACK handling to S3 and SQS inputs

[faec]

Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This can best be tested by ingesting data from a live S3 or SQS queue. The scenario that most highlights the changed performance is ingesting many small individual objects.

Related issues

• Closes aws-s3 input workers shouldn't wait for objects to be fully ingested before starting the next object #39414

[jlind23]

@faec is this ready to be reviewed by O11y team?

[faec]

@faec is this ready to be reviewed by O11y team?

I think we've talked directly since this ping, but for visibility: I don't consider this ready for review until I can try it on a live SQS queue, which has been deferred for a few reasons (previously SDH rotation and illness, currently the OTel remote-offsite). I expect that running on a live queue will immediately fail in some obvious fixable ways and I want to get those out of the way before proper review.

You also asked me to integrate #39709 (which was never merged to main) with this PR before finalizing, which hasn't been started and will probably take a day or two beyond the basic smoke check.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b awss3-ack-handling upstream/awss3-ack-handling
git merge upstream/main
git push upstream awss3-ack-handling

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[pierrehilbert]

Let's separate the work here, I added issues in your next sprint to take care of #39718

[jlind23]

Thanks @faec for the reply which close some of the knowledge gaps on my end.
I let you focus on the OTel related workshop and will come back in a few days.
The target remains unchanged and we should try to make all of those changes land before 8.16 feature freeze.

[cmacknz]

What impact will this have for users that were relying on max_number_of_messages to try to tune the input?

Is it just going to perform better with no user intervention? Could there be unintended side effects, like agents running out of memory because they can now fill up their queues?

Under what circumstances would number_of_workers be something a user has to adjust?

[faec]

What impact will this have for users that were relying on max_number_of_messages to try to tune the input?

For users that set the field to something unreasonably high to improve throughput (which was previously the only way to work around this bottleneck), they will instead get good performance by default with no added cost. The only scenario that should be "bad" in this change is if the input objects are very small (the worst-case bottleneck) and they intentionally tuned max_number_of_messages low so that ingestion would be extremely slow. In that case, ingestion speed will significantly increase (which costs a fair amount more network bandwidth, and marginally more CPU). [Though "I used this special config value to intentionally throttle my ingestion to one event per second" is very much in the "holding the escape key down to warm up the room" category imo -- total ingestion costs should not go up, it should just empty the initial queue faster.]

agents running out of memory because they can now fill up their queues?

It isn't impossible for this to happen -- if users configured a queue larger than they actually used, and were relying on the fact that this input never has more than 5-10 active events, then the new version will use more memory. However, even a full queue is usually less than 10% of Filebeat's memory footprint, so this wouldn't be a significant factor unless they had explicitly configured a very large queue that they never use.

Under what circumstances would number_of_workers be something a user has to adjust?

If they have a cap on network resources, or to a lesser extent CPU ("number of workers" is also "number of potentially parallel AWS API calls/downloads", and also number of downloaded json blobs parsed in parallel). Or in the other direction, I suppose, if they have a lot of cores and bandwidth and really want the SQS queue to move fast.

[cmacknz]

Thanks, let's add a little bit more context around this.

I think this changelog is a bit too modest, it should specifically mention the performance improvement and that tuning max_number_of_messages should not be necessary anymore.

You should also mention the small object use case and the potential to increase memory usage.

The changelogs here are usually terse but that isn't actually a requirement. You could add an entire sub-section if you wanted to.

[faec]

Done

[cmacknz]

Looks great, thanks!