how to set log stream name as pod_name and container name #16

ld-singh · 2019-09-12T02:15:15Z

Hi
How can i set log_stream_name as <pod_name><container_name><namespace_name> format instead of one we get with prefix like kube.var.log.containers?

Or How can i remove the prefix kube.var.log.containers from the stream name?

ld-singh · 2019-09-12T02:36:39Z

Can we have something like Tag_Regex for log stream also or some way to use Tag value as it is generated with help of Tag_Regex as the value for log_stream_name?

swibrow · 2019-09-30T21:02:20Z

Also looking for a way to implement this.

swibrow · 2019-10-06T13:25:53Z

@ld-singh this should do what you want

[INPUT]
        Name             tail
        Path             /var/log/containers/*.log
        Parser           docker
        Tag              kube.<namespace_name>.<pod_name>.<container_name>.<docker_id>.log
        Tag_Regex        (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
        DB               /var/log/flb_kube.db
        Refresh_Interval 5
        Mem_Buf_Limit    5MB
        Skip_Long_Lines  On

[FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.
        Regex_Parser        k8s-custom-tag
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

[OUTPUT]
         Name              cloudwatch
         Match             *
         log_group_name    fluent-bit-cloudwatch
         auto_create_group true
         region            us-east-1

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        k8s-custom-tag
        Format      regex
        Regex       ^(?<namespace_name>[^_]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?<container_name>.+)\.(?<docker_id>[a-z0-9]{64})\.log$

kerma · 2020-01-24T12:21:49Z

@swibrow your example does not work as the plugin requires either log_stream_name or log_stream_prefix to be set.
https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/blob/master/cloudwatch/cloudwatch.go#L120

I actually think it's a bug, as there is no way to set the log_stream_name to tag value. Fluent-bit doesn't support tag variable.

Default expected behavior would be to set the log_stream_name to tag value. I can submit a PR if the maintainers agree with this.

renanqts · 2020-04-24T09:18:45Z

This feature would be great. I'm thinking about how I can work out it at this moment.

tmeneau · 2020-06-02T16:24:53Z

While not ideal for solving this problem, I was able to use the nifty new tag-rewrite filter introduced in fluent-bit 1.4 to customize how the CloudWatch plugin routes logs into distinct log streams. This works since the CloudWatch plugin currently determines each event's log stream based on its tag.

# =====
# This uses two distinct tag patterns to effectively create two
# processing streams:
#
#   1. the first stream aggregates kubernetes logs into events
#      tagged with the `kube.*` pattern;
#
#   2. the second stream uses the rewrite_tag filter to copy
#      each `kube.*` event, but tags the copy with a non-overlapping
#      pattern (`namespaces.*`) that matches the desired CloudWatch
#      log stream organization. In this example the rewrite_tag filter
#      is configured to drop the original kube.* event.
#
#      Careful: the rewrite_tag filter triggers the copied event from
#      the start of the pipeline and will be reprocessed by any sections
#      that have a wildcard Match (*)!
# =====


# =====
# Processing Stream 1 (kube.*): gather kubernetes logs
#
# Gather all container logs from each node. Any modifications
# you want to make to the streams need to
# =====
[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc.cluster.local:443
    Merge_Log           On
    Keep_Log            Off


# =====
# Processing Stream 2 (namespaces.*): customize CloudWatch log stream routing
#
# Use the `rewrite_tag` filter to customize how the CloudWatch plugin routes
# events into log streams. In this case, events are grouped into streams based
# on their kubernetes namespace name.
#
# CAUTION: the rewrite_tag filter triggers an entirely new event that will
# be re-processed by the entire pipeline!
#
# see: https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag
# =====
[FILTER]
    Name                rewrite_tag
    Match               kube.*
    # the `false` at the end of rule drops the original `kube.*` event.
    Rule                $kubernetes['namespace_name'] ^(.*)$ namespaces.$1 false

[OUTPUT]
    Name              cloudwatch
    # Note: this will capture all events, but shouldn't capture the 
    # original kube.* events since they're dropped by the rewrite_tag 
    # filter.
    Match             *
    region            us-east-1
    log_group_name    fluent-bit-cloudwatch
    log_stream_prefix your-log-stream-prefix.
    auto_create_group true

PettitWesley · 2020-09-14T03:20:15Z

David's recent contributions address this feature request; we will do a release for it in the next few days.

lonnix · 2020-09-24T20:29:00Z

Has this gone out yet? I am using the cloudwatch_logs plugin and the amazon/aws-for-fluent-bit:2.6.1 image (8 days old as of this post) and I am unable to get a variable log_group_name or log_stream_name. I looked at the PR mentioned above and it mentioned #(...) as the delimiter for a variable, but I keep getting something like this:

[2020/09/24 20:25:56] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup API responded with error='InvalidParameterException', message='1 validation error detected: Value '#(fvr_log_group)' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+'

I've tried $(...) as well and get the same issue.

If this has gone out, is there documentation for how things now work?

PettitWesley · 2020-09-24T20:45:36Z

@lonnix This was released in AWS for Fluent Bit 2.7.0

Documentation is in the readme: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit

lonnix · 2020-09-24T20:51:16Z

I just updated to 2.7.0. According to that documentation this should work, specifically from the line "This value allows a template in the form of $(variable) where variable is a map key name in the log message."

    [OUTPUT]
      Name              cloudwatch_logs
      Match             kube.*
      region            us-east-1
      log_group_name    $(fvr_log_group)
      log_stream_name   $(fvr_log_stream)
      auto_create_group on
      log_format        json

but I get this in the logs

2020/09/24 20:52:51] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group $(fvr_log_group)
[2020/09/24 20:52:51] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup API responded with error='InvalidParameterException', message='1 validation error detected: Value '$(fvr_log_group)' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+'

PettitWesley · 2020-09-24T21:48:14Z

@lonnix Saw your ping in the fluent slack... we can discuss here.

So if you do $(fvr_log_group) then there must be a field in your log message which has the key fvr_log_group.

Referencing this k8s tutorial: https://aws.amazon.com/blogs/containers/kubernetes-logging-powered-by-aws-for-fluent-bit/

With k8s metadata added by fluent bit, logs might look like:

{
   "kubernetes":{
      "annotations":{
         "fluentbit.io/parser":"nginx",
         "kubernetes.io/psp":"eks.privileged"
      },
      "container_hash":"0e61b143db3110f3b8ae29a67f107d5536b71a7c1f10afb14d4228711fc65a13",
      "container_name":"nginx",
      "docker_id":"b90a89309ac90fff2972822c66f11736933000c5aa6376dff0c11a441fa427ee",
      "host":"ip-10-1-128-166.us-east-2.compute.internal",
      "labels":{
         "app":"nginx",
         "pod-template-hash":"5468c5d4d7"
      },
      "namespace_name":"default",
      "pod_id":"198f7dd2-2270-11ea-be47-0a5d932f5920",
      "pod_name":"nginx-5468c5d4d7-n2swr"
   },
   "log":"10.1.128.166 - - [19/Dec/2019:17:41:12 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"hey/0.0.1\" \"52.95.4.2\"\n",
   "stream":"stdout",
   "time":"2019-12-19T17:41:12.70630778Z"
}

So $(kubernetes['pod_name']) should be the pod name. Try that. See how it works?

[2020/09/24 20:52:51] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup API responded with error='InvalidParameterException', message='1 validation error detected: Value '$(fvr_log_group)' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+'

I think we need better error handling for this code- can you open a separate issue for that?

PettitWesley · 2020-09-24T21:56:19Z

I bet you want something like:

log_group_name /eks/$(kubernetes['namespace_name'])/$(kubernetes['pod_name'])

Everything must be spelled correctly and must be the exact right name of course..

lonnix · 2020-09-24T21:56:24Z

Using this config

    [OUTPUT]
      Name              cloudwatch_logs
      Match             kube.*
      region            us-east-1
      log_group_name    $(kubernetes['pod_name'])
      log_stream_name   $(kubernetes['pod_name'])
      auto_create_group on
      log_format        json

I tried $(kubernetes['pod_name']) and got

[2020/09/24 21:52:56] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup API responded with error='InvalidParameterException', message='1 validation error detected: Value '$(kubernetes['pod_name'])' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+'

In my example mentioned earlier I was using this log (I got this by changing the OUTPUT to stdout instead of cloudwatch_logs)

{
  "date": 1600899789.463213,
  "log": "I0923 22:23:09.463036       1 proxier.go:793] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it\n",
  "stream": "stderr",
  "time": "2020-09-23T22:23:09.46321274Z",
  "kubernetes": {
    "pod_name": "kube-proxy-tf655",
    "namespace_name": "kube-system",
    "pod_id": "998035a6-bb4f-4806-9f08-063530d418ac",
    "labels": {
      "controller-revision-hash": "6fd5694fd7",
      "k8s-app": "kube-proxy",
      "pod-template-generation": "1"
    },
    "annotations": {
      "kubernetes.io/psp": "eks.privileged"
    },
    "host": "ip-10-200-74-219.ec2.internal",
    "container_name": "kube-proxy",
    "docker_id": "09f5d9d8c4de402a40b5208dc826ee7e0f38e28ace674be965c9a4b816755960",
    "container_hash": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy@sha256:dec3e952d45de3c3eeb9a981a6724a82b83e9620585fa521cfc186dc42bd2fb8",
    "container_image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.16.8"
  },
  "fvr_log_group": "lonnix",
}

Here you can see that fvr_log_group is present but the cloudwatch_logs plugin isn't working.

PettitWesley · 2020-09-24T21:56:56Z

@lonnix Are you using 2.7.0?

lonnix · 2020-09-24T21:57:43Z

I'm using a DaemonSet with image: amazon/aws-for-fluent-bit:2.7.0

PettitWesley · 2020-09-24T21:57:56Z

CC @davidnewhall

lonnix · 2020-09-24T22:00:18Z

I tried tags 2.6.1, 2.7.0 and latest and got the same behavior on every one

LukaszRacon · 2020-09-24T23:09:36Z

Are you using correct plugin cloudwatch vs cloudwatch_logs?

PettitWesley · 2020-09-24T23:13:08Z

Good call... the go plugin is cloudwatch, the C plugin (which lacks the feature) is cloudwatch_logs.

I'm very busy right now trying to get out_s3 working for the 1.6 release of Fluent Bit... will try to test this out myself in a bit...

davidnewhall · 2020-09-24T23:41:37Z

I test this, and it seems to be OK.


func TestIssue16(t *testing.T) {
	template := "$(kubernetes['pod_name']).$(fvr_log_group)"
	nv := map[interface{}]interface{}{
		"fvr_log_group": "lonnix",
		"kubernetes":    map[interface{}]interface{}{"pod_name": "kube-proxy-tf655"},
	}
	s, err := parseDataMapTags(&Event{Record: nv, Tag: "syslog.0"}, []string{"syslog", "0"}, template)

	assert.Nil(t, err, err)
	assert.Equal(t, "kube-proxy-tf655.lonnix", s)
}

PettitWesley · 2020-09-25T07:36:29Z

So setting up a k8s cluster would take me more time than I have free right now... I created a simple logger that just spits out the full JSON event that @lonnix posted over and over again. I used a fluent bit config that parses the incoming log as JSON and then sends it to the CW output.

It works:

$ docker run -it -p 24224:24224 -v $(pwd):/fluent-bit/etc -v $HOME/.aws:/home/.aws -e "HOME=/home" amazon/aws-for-fluent-bit:2.7.0
AWS for Fluent Bit Container Image Version 2.7.0
Fluent Bit v1.5.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

INFO[0000] [cloudwatch 0] plugin parameter log_group = '$(fvr_log_group)'
INFO[0000] [cloudwatch 0] plugin parameter log_stream_prefix = ''
INFO[0000] [cloudwatch 0] plugin parameter log_stream_name = '/eks/$(kubernetes['namespace_name'])/$(kubernetes['pod_name'])'
INFO[0000] [cloudwatch 0] plugin parameter region = 'us-east-1'
INFO[0000] [cloudwatch 0] plugin parameter log_key = ''
INFO[0000] [cloudwatch 0] plugin parameter role_arn = ''
INFO[0000] [cloudwatch 0] plugin parameter new_log_group_tags = ''
INFO[0000] [cloudwatch 0] plugin parameter log_retention_days = '0'
INFO[0000] [cloudwatch 0] plugin parameter endpoint = ''
INFO[0000] [cloudwatch 0] plugin parameter sts_endpoint = ''
INFO[0000] [cloudwatch 0] plugin parameter credentials_endpoint =
INFO[0000] [cloudwatch 0] plugin parameter log_format = ''
[2020/09/25 07:33:52] [ info] [sp] stream processor started
INFO[0001] [cloudwatch 0] Log group lonnix already exists
INFO[0001] [cloudwatch 0] Created log stream /eks/kube-system/kube-proxy-tf655 in group lonnix

PettitWesley · 2020-09-25T07:41:12Z

[2020/09/24 21:52:56] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup API responded with error='InvalidParameterException', message='1 validation error detected: Value '$(kubernetes['pod_name'])' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+'

I really do not think you are using the right version somehow...

If I use a config where the keys don't exist:

[OUTPUT]
    Name cloudwatch
    Match   **
    region us-east-1
    log_group_name $(doesnotexit)
    log_stream_name /eks/$(kubernetes['thiskeyisnthere'])/$(kubernetes['nope'])
    auto_create_group true

The code seems to remove the $() special characters and just use the key name instead:

INFO[0013] [cloudwatch 0] Created log group doesnotexit
INFO[0013] [cloudwatch 0] Created log stream /eks/thiskeyisnthere/nope in group doesnotexit

lonnix · 2020-09-25T15:03:08Z

I was using the wrong plugin, I switched to cloudwatch and that solved it. Any idea if the C one is ever going to get this functionality since its supposed to be more performant and the docs at https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch say

This is the documentation for the core Fluent Bit CloudWatch plugin written in C. It can replace the aws/amazon-cloudwatch-logs-for-fluent-bit Golang Fluent Bit plugin released last year. The Golang plugin was named cloudwatch; this new high performance CloudWatch plugin is called cloudwatch_logs to prevent conflicts/confusion. Check the amazon repo for the Golang plugin for details on the deprecation/migration plan for the original plugin.

Thanks for the help!

PettitWesley · 2020-09-25T17:33:44Z

Any idea if the C one is ever going to get this functionality since its supposed to be more performant

Possibly. Though it might use a different syntax, since core Fluent Bit has some code to do this logic that unfortunately uses different templating.

Our work will be based on actual customer need and feedback. Right now, I've gotten a very small number of complaints (like 3 customers total) about max throughput limitations in the Go plugins. The go plugins do have limitations but they seem to be good enough for the vast majority of folks. Let us know if you experience any performance limitations that impact your use cases. The go plugins are easier to maintain and write features for, so new features will probably always come out in them first, and then be ported to the C plugins based on request.

lonnix · 2020-09-25T17:51:01Z

Sounds good to me. I was having issues with the fluentd cloudwatch plugin so as long as those don't show up anymore I'll be good. I know for sure we won't have throughput issues, we're not big enough for that yet :)

hencrice added the question Further information is requested label Sep 30, 2019

davidnewhall mentioned this issue Aug 13, 2020

Support dynamic log stream name from log templating variables. #78

Closed

PettitWesley added enhancement New feature or request and removed question Further information is requested labels Sep 14, 2020

PettitWesley closed this as completed Sep 24, 2020

PettitWesley reopened this Sep 24, 2020

PettitWesley mentioned this issue Sep 25, 2020

Fallback log group and stream names #99

Closed

PettitWesley closed this as completed Sep 25, 2020

ivababukova mentioned this issue Mar 22, 2022

The logging config that I used for sending fluentbit logs to Cloudwatch hms-dbmi-cellenics/iac#291

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to set log stream name as pod_name and container name #16

how to set log stream name as pod_name and container name #16

ld-singh commented Sep 12, 2019

ld-singh commented Sep 12, 2019 •

edited

Loading

swibrow commented Sep 30, 2019

swibrow commented Oct 6, 2019

kerma commented Jan 24, 2020

renanqts commented Apr 24, 2020

tmeneau commented Jun 2, 2020

PettitWesley commented Sep 14, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020 •

edited

Loading

PettitWesley commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

LukaszRacon commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

davidnewhall commented Sep 24, 2020

PettitWesley commented Sep 25, 2020

PettitWesley commented Sep 25, 2020

lonnix commented Sep 25, 2020

PettitWesley commented Sep 25, 2020

lonnix commented Sep 25, 2020

how to set log stream name as pod_name and container name #16

how to set log stream name as pod_name and container name #16

Comments

ld-singh commented Sep 12, 2019

ld-singh commented Sep 12, 2019 • edited Loading

swibrow commented Sep 30, 2019

swibrow commented Oct 6, 2019

kerma commented Jan 24, 2020

renanqts commented Apr 24, 2020

tmeneau commented Jun 2, 2020

PettitWesley commented Sep 14, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020 • edited Loading

PettitWesley commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

lonnix commented Sep 24, 2020

LukaszRacon commented Sep 24, 2020

PettitWesley commented Sep 24, 2020

davidnewhall commented Sep 24, 2020

PettitWesley commented Sep 25, 2020

PettitWesley commented Sep 25, 2020

lonnix commented Sep 25, 2020

PettitWesley commented Sep 25, 2020

lonnix commented Sep 25, 2020

ld-singh commented Sep 12, 2019 •

edited

Loading

lonnix commented Sep 24, 2020 •

edited

Loading