[filebeat] Add multiline support for journald input

[gallardo]

Describe the enhancement:

Add multiline support for journald input.

According to #27578 (comment), journalbeat has been deprecated because filebeat can now read the journal and it should support multiline. In the documentation https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-journald.html, there is unfortunately no explicit mention to multiline, and after many tests I couldn't find any working configuration so I'm assuming that this is not yet supported. This has also been suggested by fgjensen: https://discuss.elastic.co/t/filebeat-inputs-journald-multiline-not-working/294422/2?u=agallardo

Describe a specific use case for the enhancement or feature:

Correctly parse java multiline exceptions.

Tested on

$ filebeat version
filebeat version 7.16.3 (amd64), libbeat 7.16.3 [d420ccdaf201e32a524632b5da729522e50257ae built 2022-01-07 00:36:57 +0000 UTC]

Detailed description

I've first tested that I have a working config for the console:

filebeat.debug-console.yml

filebeat.inputs:
- type: stdin

  enabled: true

  multiline:
    type: pattern
    pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
    negate: false
    match: after

output.console:
  enabled: true

and an arbitrary stacktrace for testing:

stacktrace.example.txt

Interrupted after 30 s
java.lang.InterruptedException: sleep interrupted
	at java.base/java.lang.Thread.sleep(Native Method)
        at many more...(Native Method)
	at java.base/java.lang.Thread.run(Thread.java:829)
Next output ...

filebeat correctly parses the stacktrace (edited for readability):

$ cat stacktrace.example.txt | filebeat -c filebeat.debug-console.yml
{"@timestamp":"2022-01-14T11:21:17.978Z","message":"Interrupted after 30 s","input":{"type":"stdin"}}
{"@timestamp":"2022-01-14T11:21:17.978Z",input":{"type":"stdin"},"message":"java.lang.InterruptedException: sleep interrupted\n\tat java.base/java.lang.Thread.sleep(Native Method)\n\tat many more...(Native Method)\n\tat java.base/java.lang.Thread.run(Thread.java:829)"}
{"@timestamp":"2022-01-14T11:21:17.978Z","message":"Next output ...","input":{"type":"stdin"}}

Now, based on the previous configuration filebeat.debug-console.yml, I have created a new one for the filebeat.input type journald:

filebeat.debug-journald.yml

filebeat.inputs:
- type: journald

  enabled: true
  include_matches:
    - SYSLOG_IDENTIFIER=XXX

  multiline:
    type: pattern
    pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
    negate: false
    match: after

# This parsers section doesn't work either:
#  parsers:
#    - multiline:
#        type: pattern
#        pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
#        negate: false
#        match: after

output.console:
  enabled: true

But I couldn't get the multiline working (edited for readability):

$ filebeat -c filebeat.debug-journald.yml & cat stacktrace.example.txt | systemd-cat -t XXX
[2] 47924
$ {"@timestamp":"2022-01-14T14:12:40.114Z","message":"Interrupted after 30 s","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","input":{"type":"journald"},"message":"java.lang.InterruptedException: sleep interrupted"}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat java.base/java.lang.Thread.sleep(Native Method)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat many more...(Native Method)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat java.base/java.lang.Thread.run(Thread.java:829)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"Next output ...","input":{"type":"journald"}}

Notice that there are 6 events, instead of the expected 3.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[gallardo]

Sorry, I'm not following: is this ticket closed?

Thank you very much for creating this issue. However, we would kindly like to ask you to post all questions and issues on the Discuss forum first. In addition to awesome, knowledgeable community contributors, core Beats developers are on the forums every single day to help you out as well. So, your questions will reach a wider audience there, and if we confirm that there is a bug, then you can reopen this issue with the new information or open a new one.

This is the first I did on Jan 14th. See https://discuss.elastic.co/t/filebeat-inputs-journald-multiline-not-working/294422. I waited 5 days for input, and only then filed this ticket. Have I created the discussion incorrectly?

[jlind23]

Sorry @gallardo I made a mistake while changing labels. I have reopened it.

[matt-maria]

We would need this function too at the Austrian Parliament!

[dosera]

Sorry @gallardo I made a mistake while changing labels. I have reopened it.

Is this now opened somewhere else?
Also highly interested in this feature.

[ypid-work]

I have reopened it.

@jlind23 You somehow missed to reopen it. Happens :) Can you reopen now.

[fransf-wtax]

@jlind23 What's the status of this? Is it fixed? If not, can this please be reopened so it's on the radar? Thanks.

[belimawr]

Hey folks, the Journald input supports the same parsers as the Filestream input, unfortunately it's not documented yet 🤦‍♂️

Here is an example config that I've just tested in our latest release 8.15.0 and main (which bring big internal changes to the Journald input):

filebeat.inputs:
  - type: journald
    id: PR-testing
    parsers:
      - multiline:
          type: count
          count_lines: 3

Here is the documentation for the parsers: https://www.elastic.co/guide/en/beats/filebeat/8.15/filebeat-input-filestream.html#_parsers

[belimawr]

I also create an issue to add the documentation: #40478