InvalidParameterException: Upload too large with CloudWatch Go plugin [fixed in 2.31.12]

[PettitWesley]

InvalidParameterException: Upload too large with CloudWatch Go plugin

This may affects you if both of following are true:

1. You are using ANY version of AWS for Fluent Bit < 2.31.12
2. You are using the cloudwatch go plugin: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins

This issue causes Fluent Bit to fail to upload data when invalid unicode characters are present.

Versions Impacted

All releases are impacted. 2.31.12 will be the first patch version.

Resolution

2.31.12 release is actively in progress in our pipeline. Check our releases page: https://github.com/aws/aws-for-fluent-bit/releases

Mitigations

1. Switch to CloudWatch C plugin: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins
2. Use pre-release images: [all versions] exec input SIGSEGV/crash due unitialized memory [fix in 2.31.12] #661 (comment)

I want to apologize again for how long it is taking to get this release out. For you and anyone else who is waiting, here are pre-release images that you can use. They're built using the same code as our pending 2.31.12 prod release, just on a fresh EC2 intance instead of our pipeline:

144718711470.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:2.31.12-pre
144718711470.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:init-2.31.12-pre

The tag with "init" in the name is our ECS init release: https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md

These images can be pulled from any AWS account.

For example, in your Dockerfile:

FROM 144718711470.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:2.31.12-pre

Or task def:

"image": "144718711470.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:2.31.12-pre"

Or with ECS CLI:

ecs-cli pull --region us-west-2 --registry-id 144718711470 144718711470.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:2.31.12-pre

Root Cause

Code which handles utf-8.

See explanation here and here:

• Fix calculating utf-8 string length of event amazon-cloudwatch-logs-for-fluent-bit#330

[PettitWesley]

https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.31.12