Meta: Improve performance and reliability of awss3 and awscloudwatch inputs

[faec]

This issue is to aggregate the implementation work happening in the Elastic Agent / Hosted Services collaboration to improve the AWS inputs.

Issues

• awscloudwatch input drops data #38918
• aws-s3 input can delete SQS entries before they're fully acknowledged #38961
• [AWS] S3 input consumes significant amount of memory integrations#9463
• aws-s3 input writes to Filebeat registry without proper synchronization #39052
• aws-s3 input skips events and slows ingestion based on object creation timestamp #39065
• aws-s3 input treats client rate limiting as permanent failure #39114
• aws-s3 input's bucket polling accumulates state in the registry #39116
• aws-s3 input workers shouldn't wait for objects to be fully ingested before starting the next object #39414
• Issues with SQS custom endpoint setup on Main #39718

PRs

• Fix awscloudwatch worker allocation #38953
• Cleanup: Replace custom goroutine shutdown listeners with shared context wrapper #38957
• Cleanup: organizing code in awss3/input.go #38958
• Cleanup: Remove unused ACK tracking in awscloudwatch input #38983
• Fix concurrency bugs that could cause data loss in the aws-s3 input #39131
• aws-s3 input: fix SQS region selection #39327
• aws-s3 input: Split S3 poller and SQS reader into explicit input objects #39353
• Add asynchronous ACK handling to S3 and SQS inputs #40699
• [Cleanup] Delete unused/buggy EventACKTracker helper #41357
• Fix handling of custom endpoints in AWS input #41504

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

@faec I see #39414 is still open, which seems like the largest remaining performance bottleneck. Just to confirm, does that still need to be resolved? Or is it covered by in progress or completed work?

CC @pierrehilbert I see #39414 isn't in a sprint, so pending feedback from Fae on its state and importance this is still unresolved so we'll need to prioritize it.

[pierrehilbert]

This meta issue is still the main priority for @faec.
When we discussed on Tuesday she mentioned she is still expecting to have the last remaining pieces done for the 8.15 FF.
But @faec will be able to update us here :-)

[strawgate]

I also believe we need to fix these before we can release #39718 to ensure we don't regress from 8.14?