Add option for multiline exec plugin output

[matej-staron]

Is your feature request related to a problem? Please describe.
The exec plugin creates separate record for each newline produced by the executed command. There seems to be no workaround for a situation where the executed command returns multiline string that needs to be processed as a single message. I understand this approach in case of tail plugin, but I fail to see the reason why exec behaves in the same manner. The command execution is isolated between given intervals, so I'd expect the output to be returned in its entirety as a single record (possibly split depending on configured buffer size). Currently, the exec plugin uses fgets to read the output, stopping on each newline (or when reaching the buffer size) and emitting new record each time.

Describe the solution you'd like
For the sake of maintaining backwards compatibility, I would welcome some kind of Multiline bool config option for exec input plugin that would disable splitting on newlines.

The usage could look something like this:

[INPUT]
    Name          exec
    Tag           exec_ls
    Command       ls /var/log
    Interval_Sec  1
    Interval_NSec 0
    Buf_Size      8mb

    Multiline     true

[OUTPUT]
    Name   stdout
    Match  *

With following output:

[2018/03/21 17:46:49] [ info] [engine] started
[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit\nXorg.0.log\nXorg.0.log.old\nanaconda.ifcfg.log\nanaconda.log\nanaconda.program.log\nanaconda.storage.log"}]

Additional context
I'm simply trying to parse output of a command that produces multiline log. Modifying the source is not an option.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[edsiper]

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!