Dots in keys cause failure in shipping to elasticsearch from firehose.

[jbeaken]

Skaffold adds skaffold specific labels to deployments when it is used to deploy e.g

app.kubernetes.io/managed-by=skaffold-v1.12.0

if another label already exists e.g.

app=app_name

then the firehost -> elasticsearch transfer fails due to an attempt to map twice, once as text, once as an object

{"attemptsMade":8,"arrivalTimestamp":1595704200773,"errorCode":"400","errorMessage":"{\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [app.kubernetes.io/managed-by]. Existing mapping for [kubernetes.labels.app] must be of type object but found text

The dots do not cause issues with Elasticsearch fluentbit plugin, as it has a Replace_Dots option

     [OUTPUT]
       Name            es
       Replace_Dots    On
...

but for firehose, this isn't the case and elasticsearch transfer fails with

This isn't a skaffold specific issue, but more the lack of the Replace_Dots functionality of the firehose plugin. Is there any plan to add a Replace_Dots type option?

The above problem is due to using the kubernetes filter

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On

[PettitWesley]

See discussion here: aws/amazon-kinesis-streams-for-fluent-bit#13 (comment)

We could add a replace dots option in the Firehose and Kinesis go plugins though. I am not against it. The ideal scenario is something in upstream Fluent Bit that works for any output. But in the short term, implementing it here would help people.

[nsvijay04b1]

Thanks , I have a same problem and i opened a support case to AWS .

replace dots option in the Firehose and Kinesis go plugins is a sensible idea in my opinion instead of waiting for fluentbit to do something on it. After all you did it for elasticsearch plugin already.

[zhonghui12]

This is offered as part of https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.9.0.