Support UUID in Log Stream and Group templates

[PettitWesley]

One of the key limitations of CloudWatch is that each logging agent must write logs to a unique log stream- if multiple agents are concurrently writing to a single log stream you will soon get sequence token errors.

In most cases, the tag, or some field in the logs can be used to uniquely identify each instance.

However, there are some niche cases where this is not the case:

• Imagine you have thousands of servers, each with applications producing log files with the names application.log and service.log. Also, for the sake of argument, assume you have no access to an instance ID or unique host name in Fluent Bit. There's nothing for each Fluent Bit instance to use to uniquely identify itself and write to a unique log stream
• Currently, we do not have an ECS Metadata filter. I have heard of cases where folks want to run Fluent Bit as a side-car and read log files created by their app container. But there is no way to uniquely identify each task and have it send to a separate log stream. This will be solved when we add support for a generic ECS Metadata filter.

Those cases are a bit contrived... but I do think this problem of "nothing uniquely identifies each fluent bit to allow it to create a different log stream" is a real problem. Not a common problem, but something worth solving if there's a simple solution.

One possible solution is to support a new special template, $(uuid) which would add a UUID to your log stream or group.

The Fluentd S3 plugin has this feature: https://github.com/fluent/fluent-plugin-s3

[PettitWesley]

CC @davidnewhall for comment (not asking you to implement this)

[PettitWesley]

Thinking about this some more, the main use case I know for this is when folks are running in ECS but their logs don't come from stdout and so the metadata FireLens provides is not useful.

I think it'd be good to just build ECS metadata injection into this plugin, since the upstream fluent bit ECS Metadata filter will take some time to build. In the meantime we can fairly quickly just add special variables ecs_task_arn and ecs_cluster into this plugins templating feature.

[PettitWesley]

Actually, let's do ecs_task_id instead

[davidnewhall]

I feel like as long as we "know the app is running in ECS" (and why wouldn't it be?) then pulling the ECS meta data (arn, id, cluster name) seems like a decent path forward to finding a unique bit of data. Creating many log streams inside one group seems perfectly normal and desirable when you have many tasks running. I can totally stand behind that ideal.

Generating and using random UUIDs may work in some cases too, but I fear opening that can of worms; I believe folks would try it, get confused and post questions asking how it works and how they make it work differently. I just think an initial implementation of randomness like that would be poorly-received. I could be wrong, but as a 20+ year programmer that's the feeling I get.

Do let me know if I provided useful insight and if you have any more questions. Good luck!