Skip to content

AWS for Fluent Bit 2.31.2
Compare
Choose a tag to compare
@matthewfala matthewfala released this 16 Feb 19:51
· 270 commits to mainline since this release
v2.31.2
20b9922

2.31.2

This release includes:

  • Fluent Bit 1.9.10
  • Amazon CloudWatch Logs for Fluent Bit 1.9.1
  • Amazon Kinesis Streams for Fluent Bit 1.10.1
  • Amazon Kinesis Firehose for Fluent Bit 1.7.1

Compared to 2.31.1, this release adds the following feature that we are working on getting accepted upstream:

Same as 2.31.1, this release includes the following fixes and features that we are working on getting accepted upstream:

  • Feature - Add kinesis_firehose and kinesis_streams support for time_key_format milliseconds with %3N option, and nanoseconds 9N and %L options fluent-bit:2831
  • Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
  • Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
  • Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
  • Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
  • Bug - Resolve S3 logic to display log_key missing warning message if the configured log_key field is missing from log payload fluent-bit:6557
  • Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505
  • Bug - Support Retry_Limit option in S3 plugin to set retries fluent-bit:6475
  • Bug - Resolve a rare Datadog segfault that occurs when remapping tags aws-for-fluent-bit:491

We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit under different input load. Learn more about the load test.

plugin source 20 MB/s 25 MB/s 30 MB/s
kinesis_firehose stdstream Log Loss ✅ ✅ ✅
Log Duplication ✅ ✅ ✅
kinesis_firehose tcp Log Loss ✅ ✅ ✅
Log Duplication 0%(500) ✅ ✅
kinesis_streams stdstream Log Loss ✅ ✅ 0%(5795)
Log Duplication 0%(500) 0%(32694) 0%(63482)
kinesis_streams tcp Log Loss ✅ ✅ ✅
Log Duplication ✅ 0%(500) 0%(6445)
s3 stdstream Log Loss ✅ 12%(1874773) 30%(5479384)
Log Duplication ✅ ✅ ✅
s3 tcp Log Loss ✅ ✅ 15%(2818466)
Log Duplication ✅ ✅ ✅
plugin source 1 MB/s 2 MB/s 3 MB/s
cloudwatch_logs stdstream Log Loss ✅ ✅ ✅
Log Duplication ✅ ✅ ✅
cloudwatch_logs tcp Log Loss ✅ ✅ ✅
Log Duplication ✅ ✅ ✅

Note:

  • The green check ✅ in the table means no log loss or no log duplication.
  • Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
  • Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.
Assets 2
Loading