Join partial docker json-file lines

[jlpettersson]

Is your feature request related to a problem? Please describe.

Fluent-bit does not join log lines that are splitted as "partial" by docker daemon, when logging to json-file. https://github.com/moby/moby/blob/master/daemon/logger/jsonfilelog/jsonfilelog.go#L135

Describe the solution you'd like

Fluent-bit should join these lines, either in the parser or in a filter.

Describe alternatives you've considered

We currently have issues with this and don't see any good workarounds.

Additional context

Log events are sent to Splunk, splitted over multiple events and unparsed since they are not considered to be valid json when splitted.

[jlpettersson]

This should be solved with a multiline parser in tail plugin

[NordbergDK]

We have the exact same challenge.
Does closing the issue mean you have found a viable solution for this using multiline parser in tail plugin?

I tried:

[INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Multiline         On
        Parser_Firstline  docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

With the default docker parser

[PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log

This does not work and still breaks into multiple entries.

@jlpettersson Any help would be greatly appreciated

[ReDim42]

We have the exact same challenge.
Does closing the issue mean you have found a viable solution for this using multiline parser in tail plugin?

I tried:

With the default docker parser

This does not work and still breaks into multiple entries.

@jlpettersson Any help would be greatly appreciated

@NordbergDK The Parser_Firstline must point uniquely to a first line of a multilog message. That means a parser as docker in the format JSON does not help here. Why? We matched every line there. Because every line contains the JSON field "log" and therefore every entry is the beginning.

For example, I have used a regex to specify the beginning of a multiline message.
Remember u need to have one named_capturing group.

Cheers

[jlpettersson]

I am working on a filter to handle partial messages from e.g. Docker and CRI-O.

[birdy81]

I am working on a filter to handle partial messages from e.g. Docker and CRI-O.

@jlpettersson @edsiper Is there already a solution available or in planning for CRI-O log format (we use containerd)? We have large logs entries in JSON format that are splitted in F (Full) and P (Partial). Therefore, the JSON cannot be parsed automatically. Fluentbit is working very good in our environment. But we need a solution that these partial logs are merged.

2019-07-02T07:46:09.000766472Z stdout P { JSON....
2019-07-02T07:46:09.00077217Z stdout F ...Rest of JSON}

[g3kr]

@jlpettersson

I am working on a filter to handle partial messages from e.g. Docker and CRI-O.

@jlpettersson Is there a solution for this today? We see the key partial set to true in these log events. Log events are sent to CloudWatch, splitted over multiple events and unparsed since they are not considered to be valid json when splitted. join_partial filter plugin - can this be used with forward input? Do you have an example for this?

Multiline Parser with in_tail is something different an works only for tail input plugin.

Your response/help is appreciated.

[tobiangerstein]

@jlpettersson @edsiper Is there anything similar to the "Docker_Mode" option for CRI-O (containerd)? In our project context, many log entries are split due to their size and cannot be analyzed properly. Fluentbit perfectly fits in our infrastructure environment, but the lack of the possibility to join partial CRI-O logs more and more becomes a showstopper for us. Do you have any suggestions, how we can solve this problem?

[simhnna]

Are there any planes on implementing this in the near future? Not a C pro, but I guess I could take a stab at it, if I know my work will get merged...

[bpottier]

Any update on this? Anxiously awaiting this feature...

[edsiper]

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

[tobiangerstein]

🥳