[Question] [Help] - Kubernetes metadata or different log_group for namespace

[midestefanis]

¿Are you planning to add Kubernetes metadata to the logs?, eg:

[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On

Or how can I collect all the logs of a namespace and send to a specific log_group. Because with the default configuration all the logs go to the same log_group. Default configuration:

[OUTPUT]
Name cloudwatch_logs
Match *
region us-east-1
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group true

Without the k8s metadata in the logs and without being able to send the logs of a namespace to a particular log_group I cannot distinguish the different applications.

[hossain-rayhan]

Hi @midestefanis , you can enrich your log with kubernetes metadata using the kubernetes filter and use a placeholder in your log_group and log_stream name. Did you try following this guideline in our README? Hope it will give some guidance.

[midestefanis]

Hi @hossain-rayhan thanks for your answer. Is the kubernetes filter working in this version of fluent-bit? And yes, i have looked into that readme but it doesn't say anything about eks, only ecs.

[PettitWesley]

@midestefanis The tutorial shows you how to generically use keys in the logs to template the stream/group names. It works with any keys.

If you have a key in the logs like kubernetes.pod_name you could make the log stream name template be:

    log_stream_name from-fluent-bit-$(kubernetes['pod_name']

[midestefanis]

Hi @PettitWesley, yes, but to do that the application has to log in with that key. In a large company you cannot make all the applications that want to mount on your infra to have to refactor their logs. The correct approach would be to be able to have the kubernetes metadata (with the kubernetes filter) and add it to it.

[PettitWesley]

@midestefanis I think you are not understanding my suggestion.

You can do the following:

1. Add Kubernetes filter to your config map. It will then add kuberenetes metadata keys to your logs. You do not have to manually edit the logs or anything like that. See this tutorial for general setup of fluent bit on EKS and also an example of the kubernetes metadata that will be added: https://aws.amazon.com/blogs/containers/kubernetes-logging-powered-by-aws-for-fluent-bit/
2. Then you update your config map with my suggestion above. The plugin will take the kubernetes metadata in the logs and generate the log stream name based on the template.

This experience works. We have many customers using it.

[midestefanis]

@PettitWesley It doesn't work on eks-fargate. I have already tried it. I found this -> aws/containers-roadmap#1197

[PettitWesley]

Oh I did not realize you are an EKS Fargate customer. In that case, yes, you must wait till that roadmap issue is implemented. We know that it is a high priority request that many customers need.

[vinay-ux]

@PettitWesley I am having trouble with this on EKS too. I can see the metadata from kubernetes but the templating does not work for some reason.

My config:

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    DB                /var/log/flb_kube.db
    Parser            docker
    Docker_Mode       On
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc.cluster.local:443
    Merge_Log           On
    Merge_Log_Key       data
    Keep_Log            On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
[OUTPUT]
    Name                  cloudwatch
    Match                 *
    region                eu-central-1
    log_group_name        eks/workloads
    log_stream_name       from-fluent-bit-$(kubernetes['namespace_name'])
    log_stream_prefix     fluentbit-$(kubernetes['pod_name'])
    log_format            json
    log_retention_days    7

{
"kubernetes": {
"annotations": {
"kubernetes.io/psp": "eks.privileged"
},
"container_hash": "quay.io/calico/node@sha256:bc4aa22272ef814ed4c3ce02b78fab8ddf446fceeef370482c4ebc4edabc06a1",
"container_image": "quay.io/calico/node:v3.19.1",
"container_name": "calico-node",
"docker_id": "ce0887fc17decdafabc392ff973db5e5057bce7c0e581edea00a59f96179f1c9",
"host": xxxxxl",
"labels": {
"app.kubernetes.io/name": "calico-node",
"controller-revision-hash": "58665c8cf4",
"pod-template-generation": "1"
},
"namespace_name": "networking",
"pod_id": "d446a251-abfa-4382-86f2-9e0150e1ef6e",
"pod_name": "calico-node-qtjpk"
},
"log": "2021-07-12 17:11:18.015 [INFO][39] felix/config_params.go 432: Parsing value for FelixHostname: xxxxxx (from environment variable)\n",
"stream": "stdout",
"time": "2021-07-12T17:11:18.015137585Z"
}

[PettitWesley]

@vinay-ux The problem might be here:

log_stream_name       from-fluent-bit-$(kubernetes['namespace_name'])
log_stream_prefix     fluentbit-$(kubernetes['pod_name'])

The templating works only with log_group_name and log_stream_name. Don't also use the prefix option, I think it takes precedence.

What is the final log stream name in CW?

@hossain-rayhan Can we add a validation so that both prefix and name are an invalid config.

[hossain-rayhan]

@PettitWesley I think I understand what you meant. Just to be sure, log_stream_name and log_stream_prefix are either / or option. They cannot be put together and we will add a validation to stop such case. I am taking an action item.

[PettitWesley]

@hossain-rayhan yeah they plugin should fail if they are used together IMO.

[vinay-ux]

Right, this makes a lot of sense @PettitWesley , thank you very much :)

What is the final log stream name in CW?
$(kubernetes['namespace_name'])-var.log.containers......

[2won0]

Dear @PettitWesley , I have a question about this issue.

I want to create log group for each namespace. But it's not working.
I checked logs from fluentbit pod, InvalidParameterException occured.

Is there something I have configured incorrectly?

My config.

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    DB  /var/log/flb_kube.db
    Parser docker
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name               cloudwatch_logs
    Match               kube.*
    region              ap-northeast-2
    log_group_name      /eks/dev/$(kubernetes['namespace_name'])
    log_stream_name     $(kubernetes['pod_name'])
    auto_create_group   true

Exception message
[2024/08/22 06:27:16] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream API responded with error='InvalidParameterException', message='1 validation error detected: Value '/eks/dev/$(kubernetes['namespace_name'])' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+'

Even though I have configured it in the form of $(kubernetes['namespace_name']) as referenced on the https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names page, it seems that the value is not being recognized. Is there anything else I can check?

[2won0]

I share how I solved.

[OUTPUT]
    Name                  cloudwatch_logs
    Match                 application.*
    region                ${AWS_REGION}
    log_group_name        fallback-group
    log_stream_prefix     fallback-stream
    log_group_template    /eks/dev/$kubernetes['namespace_name']
    log_stream_template   $kubernetes['pod_name']
    auto_create_group     true