multiline match is not parsed by JSON and kubernetes

[kdihalas]

Bug Report

Describe the bug
When i use the multiline parser the matched log field is escaped and cannot be parsed furthermore as a struct by the docker parser. I tried escaped, escaped_utf8, json nothing works. Because of that kubernetes filter cannot merge the log to the root level of the struct. Is there something that i am missing here?

To Reproduce
https://rubular.com/r/0vjruc4o6AG8la

{
	"log": "{\"@timestamp\":\"2019-10-30T09:00:45.802Z\",\"application\":\"air-travelfusion\",\"build_number\":\"1.1.25\",\"build_timestamp\":\"20191025-105856\",\"certificate_name\":\"\",\"correlation_id\":\"XXXXXXXXXXXX\",\"cte\":{\"category\":\"Service\",\"subcategory\":\"Response\"},\"data_version\":2,\"description\":\"Service Response HTTP Status: 200\",\"endpoint\":\"/v1/AirSearchBySchedule\",\"host\":\"XXXXXXXXXXX\",\"level\":\"TRACE\",\"login_id\":\"XXXXXXXXXX\",\"method\":\"XXXXXXXXX\",\"name\":\"\",\"response_obj\":{\"content\":\"\\u003cCSResponse\\u003e\\u003cResponseXML\\u003e\\u0026lt;AirAvailResp hasFaresByLeg=\\u0026#34;0\\u0026#34; concur-correlationid=\\u0026#34;XXXXXXXXX\\u0026#34;\\u0026gt;\\u0026lt;/AirAvailResp\\u0026gt;\\u003c/ResponseXML\\u003e\\u003c/CSResponse\\u003e\",\"status_code\":200},\"roletype\":\"airtf\",\"type\":\"log\"}\n",
	"stream": "stderr",
	"time": "2019-10-30T09:00:45.802288564Z"
}

• Run with the example config
• You should see errors that the log field is not JSON and cannot be parsed

Expected behavior
Named regex must return a JSON not an escaped string.
JSON Parser should be able to parse the matched string.

Your Environment

• Version used: 1.2
• Configuration: https://gist.github.com/kdihalas/b9f71eb70312ce15c2487d30c52b3dc2
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version: Docker 1.2
• Operating System and version: distroless docker
• Filters and plugins: kubernetes

Additional context
Generated log:
{"date":1572435032.302837,"log":"{\\\"@timestamp\\\":\\\"2019-10-30T09:00:55.591Z\\\",\\\"application\\\":\\\"ADADADADADA\\\",\\\"build_number\\\":\\\"1.1.25\\\",\\\"build_timestamp\\\":\\\"20191025-105856\\\",\\\"certificate_name\\\":\\\"\\\",\\\"correlation_id\\\":\\\"ADADADADADADADA\\\",\\\"cte\\\":{\\\"category\\\":\\\"External\\\",\\\"subcategory\\\":\\\"Request\\\"},\\\"data_version\\\":2,\\\"description\\\":\\\"\\\",\\\"endpoint\\\":\\\"/v1/ADADDADAA\\\",\\\"host\\\":\\\"ADADADADADAAD\\\",\\\"level\\\":\\\"TRACE\\\",\\\"login_id\\\":\\\"ADADADADADADAD\\\",\\\"method\\\":\\\"ADADADADADAD\\\",\\\"name\\\":\\\"v1/ADADDADAD\\\",\\\"request_obj\\\":{\\\"body\\\":\\\"\\\",\\\"method\\\":\\\"v1/ADADADADADAD\\\",\\\"path\\\":\\\"ADADADADADA/v1/ADADADADA\\\",\\\"query\\\":\\\"\\\"},\\\"roletype\\\":\\\"airtf\\\",\\\"type\\\":\\\"log\\\"}\\n","type":"log","data_version":"2","roletype":"k8s","application":"kubernetes","cluster":"XXXXX"}

[edsiper]

(this is not longer an issue, but posting our new Multiline updates)

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!