Fluent-bit configuration for multiline.parser stops log forwarding

[karanpreetsingh1990]

Description

Including multiline.parser in the configuration for logging stops forwarding logs to New Relic but Tail plugin works without it.

[Troubleshooting]

As a part of troubleshooting I tried running the fluent-bit.exe plugin directly and found that it wasn't capturing the changes to the file as they were added.

Removing the multiline.parser option seemed to correct this behaviour.

I also noticed that the same config when running locally was working as expected. The only difference was the version of fluent-bit.

The infra agent currently includes v1.9.3 but I had v2.2.1 on my machine. As a part of troubleshooting I replaced the fluent-bit.exe and fluent-bit.dll files in the newrelic-integrations\logging directory and the same configuration was now working as expected.

Steps to Reproduce

Fluent-bit configuration file:

[SERVICE]
    flush        1
    log_level    trace
    parsers_file parsers.conf

[INPUT]
    Name tail
    Path testfile.txt
    Path_Key filePath
    Key message
    Tag multiline_logs
    Mem_Buf_Limit 100MB
    multiline.parser multiline_regex_test

[FILTER]
    Name parser
    match multiline_logs
    key_name message
    parser named-capture-test
    Reserve_Data On

[FILTER]
    Name record_modifier
    Match multiline_logs
    Record logtype testlog
    Record application test

Parsers file:

[MULTILINE_PARSER]
    name          multiline_regex_test
    type          regex
    key_content message
    flush_timeout 1000
    # Regex rules for multiline parsing
    # ---------------------------------
    #
    # configuration hints:
    #
    #  - first state always has the name: start_state
    #  - every field in the rule must be inside double quotes
    #
    # rules   |   state name   | regex pattern                   | next state name
    # --------|----------------|--------------------------------------------------
    rule      "start_state"   "/^(\d{4}-\d{2}-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}) (.*)/"    "cont"
    rule      "cont"          "/^[^\d].+$/"                       "cont"

[PARSER]
    name named-capture-test
    format regex
    Regex /^(?<date>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) (?<message>.*)/m

Log file test entries

2023-11-24 05:52:22.201 DEBUG Class :'test',Method :'execute',Message :Logger initialized8
2023-11-24 05:52:22.201 DEBUG Class :'test',Method :'execute',Message :Logger initialized7
asdfasdf
adsfasdfasdf
asdfasdfasdf
asdfasdfsgdfh
dfsghdfghb
sdfsfhdh
2023-11-24 05:52:22.201 DEBUG Class :'test',Method :'execute',Message :Logger initialized8

Your Environment

NR infra-agent version: 1.48.1
OS: Windows 2019 server
Fluent-bit plugin version: 1.9.3

Additional context

I tried finding a reference to this in the fluent-bit issues but didn't find this. I believe this may be resolved by upgrading the version of fluent-bit bundled with the infrastructure agent.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-200621

[jdearana-nr]

@karanpreetsingh1990 We recommend you to upgrade to the latest infra-agent version, which currently bundles FluentBit 3.0.4. The issue you're experiencing seems to be related to this issue Multiline filters stopped working in 1.9.3 · Issue #5524 · fluent/fluent-bit, which is fixed in FluentBit 1.9.6 and onwards. TY!

[workato-integration]

This issue won't be actioned.

[jdearana-nr]

@karanpreetsingh1990 For Windows, NR's next infra-agent release will include FluentBit 3 that also fixes the issue.