The logging config that I used for sending fluentbit logs to Cloudwatch #291
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| kind: Namespace | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: aws-observability | ||
| labels: | ||
| aws-observability: enabled | ||
| --- | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: aws-logging | ||
| namespace: aws-observability | ||
| data: | ||
| # NEVER set this to true, or set only temporary | ||
| # it enables internal fluentbit logs that flood Cloudwatch | ||
| # and cause huge AWS bills | ||
| # (see slack post https://this-is-biomage.slack.com/archives/C014YMUT6GN/p1646915799716329) | ||
| flb_log_cw: "false" | ||
| output.conf: | | ||
| [OUTPUT] | ||
| Name cloudwatch | ||
| Match * | ||
| region eu-west-1 | ||
| log_group_name fluent-bit-cloudwatch | ||
| log_stream_name $(kubernetes['pod_name']) | ||
| auto_create_group true | ||
|
|
||
| parsers.conf: | | ||
| [PARSER] | ||
| Name docker | ||
| Format json | ||
| Time_Key time | ||
| Time_Format %Y-%m-%dT%H:%M:%S.%L | ||
| Time_Keep On | ||
|
|
||
| filters.conf: | | ||
| [FILTER] | ||
|
There was a problem hiding this comment. I didn't try this, but it looks useful for adding some more metadata to the logs: https://docs.fluentbit.io/manual/pipeline/filters/aws-metadata |
||
| Name kubernetes | ||
| Match * | ||
| Merge_Log On | ||
| Buffer_Size 0 | ||
| Kube_Meta_Cache_TTL 300s | ||
| Labels On | ||
| K8S-Logging.Exclude On | ||
|
Comment on lines
+38
to
+44
There was a problem hiding this comment. This adds to the logs the kubernetes pod details, so that later on we can add pod name as log_stream_name. This will cause a separate stream to get created for each pod and sending the logs to it. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [{ | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "logs:CreateLogStream", | ||
|
There was a problem hiding this comment. we need permission to allow the logs to set retention time. We also need to complete this permissions json to create a new permission. Right now, I created a new permission from the AWS console, called "eks-fargate-logging-policy" and manually attached it to the Fargate pod execution roles. There was a problem hiding this comment. After the role is created properly, this role created by the console needs to be deleted |
||
| "logs:CreateLogGroup", | ||
| "logs:DescribeLogStreams", | ||
| "logs:PutLogEvents" | ||
| ], | ||
| "Resource": "*" | ||
|
There was a problem hiding this comment. this * needs to be changed to something less generic. |
||
| }] | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here I use the non-official Fluent Bit output plugin: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit. The official plugin is called "cloudwatch-logs" and it doesn't work for us, because it doesn't support templating variables (what we do to get the log_stream_name equal to the pod name).
Link to very useful github issue: aws/amazon-cloudwatch-logs-for-fluent-bit#16
Link to "cloudwatch_logs" that we didn't use in the end: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch
This config used to work, but it needs to be double-checked because it started failing at some point. There is a chance that it's not working