INFO: AWS Distro differences for S3

[PettitWesley]

Currently, there are a number of features/changes for upstream Fluent Bit S3 that have been reverted in the AWS Distro. 2.31.3 was most recent release with all recent S3 changes:

• https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.31.3

Specifically:

• Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
• Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
• Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475

These changes were removed since our stability tests associated them with an increase test failures (crashed/stopped tasks).

All S3 enhancements will return as part of the upcoming S3 stability fix release, which is work in progress:

• out_s3: daemon thread stability refactor rebased on top of AWS distro 2.31.10 PettitWesley/fluent-bit#24
• Fluent Bit Scheduled Timer with output coroutine design fluent/fluent-bit#7466

Retries in S3

Currently in the AWS Distro S3 does not support Retry_Limit and retries are hardcoded at 5 retries.

Currently when an upload is dropped due to failed retries, you get messages like these:

" Upload for {s3 key} has reached max upload errors"

"Chunk file failed to send %d times, will not "retry"

"Upload for %s has reached max completion errors, plugin will give up"

"File with tag %s failed to send %d times, will not retry"