feature request: cloudwatch_logs templating failure can lead to a very large number of warn logs

[PettitWesley]

See: fluent/fluent-bit#6918

The warning for record_accessor failure here can be emitted for every single log record processed: https://github.com/fluent/fluent-bit/blob/master/plugins/out_cloudwatch_logs/cloudwatch_api.c#L1084

It'd be ideal if this was only emitted once per batch instead. This will require code changes.

[PettitWesley]

For 2.0 there are workarounds: fluent/fluent-bit#6918 (comment)

[PettitWesley]

This is a concern for the upcoming Daemon support launch, since it makes extensive use of templating which can fail .

[Mattie112]

Yes we also had this issue costing quite a bit (and yes we now have alerts). Would be great to have a solution indeed!

[petemounce]

It's not only templating that can fail with this impact, but also access denied when attempting to (say) create a log stream (where fluent bit can create the log stream for its own logs for whatever misconfiguration reason).

[PettitWesley]

The new log suppress feature in 2.0 is good for the CW access denied and other errors. Which suppresses the same message from a single plugin.

For the templating failures, the message actually comes from a core library which I think is not part of the log suppress feature.

[lexsca]

we ran into this as well and racked up massive cloudwatch changes. the default values should not have any template variables that aren't guaranteed to be there. people will probably keep tripping over this. 😐

[PettitWesley]

When Fluent Bit is deployed as a kubernetes daemonset pod, it collects its own logs, and thus, when it emits an error message, it will collect its own error message, potentially leading to a cycle of log spam in which each of its own error logs cause it to produce another error log.

Key cases:

1. This issue- templating failure.
2. Log stream creation failure or API failure in general: if FLB can not create log streams or uploads, then this will lead to error logs, and then more error logs for the failure to upload those error logs.

[PettitWesley]

See one option for workaround here: fluent/fluent-bit#6918 (comment)