Use of log_key parameter silently fails when the value is not a string

[fs-alecbg]

Setting the log_key value to log in the Output for Fluent-bit does work. However, if one uses a nest filter to create an object called for_cloudwatch whose value is a map, then setting log_key to for_cloudwatch will fail to write any logs and not print any errors (except for debug logs saying that it has received empty logs).

For example, if we have set

    [SERVICE]
        # sync to outputs every 1 second
        flush 1 
        Parsers_File /fluent-bit/parsers/parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        # The fluent-bit input of type "tail" should use the filesystem to buffer log entries, not just memory.
        storage.type  filesystem
        
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Merge_Log_Key       data
        Keep_Log            On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Buffer_Size         32k
 
    [FILTER]
        Name                nest
        Match               *
        Operation           nest
        Wildcard            log
        Wildcard            stream
        Nest_under          for_cloudwatch
       
    [OUTPUT]
        Name                  cloudwatch
        Match                 *
        region                my-region
        log_group_name        /aws/eks/my-cluster/containers/$(kubernetes['namespace_name'])
        log_stream_name       $(kubernetes['pod_name'])/$(kubernetes['container_name'])
        log_key               for_cloudwatch
        auto_create_group     true
        storage.total_limit_size 500M

This configuration will write no logs to CloudWatch and Fluent-bit's logs will show debug logs saying that the logs are empty.

There are two bugs in two codebases causing this issue.

The reason that there are no error logs is because the err here is shadowed in the inner scope and thus when we get to the point where the err should be handled, it is nil. The value of data at this line is an empty array and this leads to this debug line being executed.

The error itself is caused by the fact that in the amazon-cloudwatch-logs-for-fluent-bit codebase this line returns an empty array and an error because the input type is interface{}.

[thamjieying]

i am facing this issue as well, are there any fix for this?