Embedded metrics in ECS task configured with awslogs Log Driver

[jhecking]

Is it possible to use this client to generate embedded metrics in an ECS task that's configured with the awslogs Log Driver? Like in a Lambda environment, the task just logs to stdout and the logs are automatically collected and send to CloudWatch. So the LambdaSink should work just as well for the ECS environment as well. The problem, I guess, is in the environment detection. Unlike the Lambda environment, I don't think it's possible to automatically detect the ECS task environment as there are no special env vars set (AFAIK). And even if that's possible, then the next challenge would be to detect whether the awslogs Log Driver is being used. So probably the configuration of the client would have to be done manually for the ECS task environment.

Would a PR for this feature likely be accepted?

[jaredcnance]

Great question @jhecking. That's mostly correct. In order to support the awslogs driver, we would need to modify the driver to include the x-amzn-logs-format header described here. Right now we have no plans to modify the driver since ECS is recommending moving to FireLens (which uses FluentBit), instead of the awslogs driver. We do plan to support EMF in the amazon-cloudwatch-logs-for-fluent-bit project soon.

From the perspective of this library, we probably just need to provide a simple setup for this scenario that allows you to explicitly configure which sink the logs are sent to without going through the environment detection step.

We won't begin work on this for another 1-2 weeks, but PRs in the meantime are certainly welcome 😃.

[jhecking]

Great. Thanks for the quick response and for the pointer to the fluent bit ticket. I'll be monitoring that. For now, we have decided to just use the CloudWatch PutMetricData API directly.

[jaredcnance]

Closing this issue since it's not specific to the client libraries. Please re-open if I missed something. We're tracking the FireLens integration here: aws/amazon-cloudwatch-logs-for-fluent-bit#42. Additionally, you can configure the CloudWatch Agent to run on ECS/EKS as described here and configure it to send embedded metrics as decribed here.

[gordonpn]

@jhecking

Any EMF logs that can reach CloudWatch Logs will now be parsed and metrics can be generated from them. You don't need the EMF header anymore.

If you face any issues, you can get in touch with me!

[anayak-cmt]

@gordonpn Does it mean we can use awslogs driver to emit EMF logs (no need to use awsfirelens log driver as was suggested in one of the earlier comments)

[gordonpn]

@gordonpn Does it mean we can use awslogs driver to emit EMF logs (no need to use awsfirelens log driver as was suggested in one of the earlier comments)

@anayak-cmt, yes that is what that means.

You need to write your EMF logs to stdout and get them to CloudWatch Logs. One of the many ways to do that is with the awslogs driver. When your EMF logs reach CWL, they will be processed and metrics will be published.

[skidson]

Any EMF logs that can reach CloudWatch Logs will now be parsed and metrics can be generated from them. You don't need the EMF header anymore.

This needs to be made clear in AWS' documentation on EMF. This solved a big headache for me trying to get EMF working for my AWS Batch application which makes sidecar containers difficult/impossible but logging to CW trivial. Sadly I spent at least a day fiddling around with CW Agent before finding this.