Support Cloudwatch Embedded Metric Format Logs

[talawahtech]

Cloudwatch now supports automatically generating metrics from logs sent in a custom format.

It would be great if this plugin provided support for sending logs to CloudWatch in the expected format.

Would love to see support for built-in fluent-bit plugins (e.g. in_cpu) as well as well as for logs coming from an application with application specific metrics.

[PettitWesley]

@talawahtech This is something we're thinking about!

Right now, there is a very simple solution available. Have your app produce logs in the format (either by constructing them yourself or using one of the CloudWatch libraries). This plugin then just needs to add a header in its http requests to note that the logs are formatted as EMF. I expect we'll get to this fairly soon- feel free to open a PR for it yourself though.

Long term, having a way to take incoming data (ex built-in cpu plugin or custom app metrics) and convert it to EMF would be ideal. However, that requires that the the input data is in some recognizable format (which we can then convert).

This is something I've been thinking about recently- how can Fluent Bit better process metrics in general when there is no single common metrics format? One idea is to just accept the lack of a common format, and add filters to fluent bit which can convert from one format to another. But that'll likely be time consuming to implement.

Curious if you have thoughts this specific problem or the more general problem.

[talawahtech]

Great to hear. I think support for the built-in metrics plugins like cpu/mem/netif/disk would be a great start. There is also an in_statsd plugin under development to support receiving metrics for statsd compatible sources, that could potentially be a next step.

I think it may also be useful to keep an eye on the upcoming OpenTelemetry Metrics API. The format they use seems like it will be more complex, but if fluent-bit ever gets an in_otmetrics plugin then that might also be a potentially area to support automatic conversion into EMF.

[cwendt94]

Is there a timeline when a new release will be published with this implementation?

[PettitWesley]

@cwendt94 Hopefully in the next 2 weeks. We need to get the integration tests fixed before we can put this out.
aws/aws-for-fluent-bit#21

[cwendt94]

@PettitWesley Sounds good, thank you for the quick response!

[PettitWesley]

@cwendt94 @talawahtech Released in 2.2.0: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.2.0