Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to read multiline json format #2418

Closed
yeruvass opened this issue Aug 4, 2020 · 6 comments
Closed

How to read multiline json format #2418

yeruvass opened this issue Aug 4, 2020 · 6 comments
Labels

Comments

@yeruvass
Copy link

yeruvass commented Aug 4, 2020

I'm trying to read below json format but reading line by line .

td-agent-bit version is 1.4.5

Log format :-

{
"thread" : "1",
"level" : "OFF",
"loggerName" : "AuditLogger",
"marker" : {
"name" : "Audit",
"parents" : [ {
"name" : "EVENT"
} ]
},
"message" : "Audit [useCredentials fileName="Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)" name="Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)" timestamp="2020-06-02T12:34:36.928+04:00" usage="[11,1864)"]",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.logging.log4j.audit.AuditLogger",
"instant" : {
"epochSecond" : 1594986664,
"nanoOfSecond" : 204000000
},
"contextMap" : { },
"threadId" : 129467,
"threadPriority" : 5
}
{
"thread" : "2",
"level" : "OFF",
"loggerName" : "AuditLogger",
"marker" : {
"name" : "Audit",
"parents" : [ {
"name" : "EVENT"
} ]
},
"message" : "Audit [useCredentials fileName="Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)" name="Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)" timestamp="2020-06-02T12:34:36.928+04:00" usage="[11,1864)"]",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.logging.log4j.audit.AuditLogger",
"instant" : {
"epochSecond" : 1594986664,
"nanoOfSecond" : 204000000
},
"contextMap" : { },
"threadId" : 129467,
"threadPriority" : 5
}

td-agent-bit.conf
[SERVICE]
Parsers_File /etc/td-agent-bit/parsers.conf
Daemon off
Flush 5
Log_Level trace

[INPUT]
Name tail
Path /var/log/jenkins/audit1.log
DB /etc/td-agent-bit/db/enbd-jenkins-audit.db
Path_Key tailed.path
Tag enbd-jenkins-audit
Parser_Firstline firstline2
Multiline On
Parser_1 multiline2
Refresh_Interval 5
Buffer_Chunk_Size 3M
Buffer_Max_Size 200M
Mem_Buf_Limit 300M
Skip_Long_Lines true
[OUTPUT]
Name stdout
Match enbd-jenkins-*

parsers.conf
[PARSER]
Name firstline2
Format regex
Regex ^(?{\n\s{1,}"thread"\s:)

[PARSER]
Name multiline2
Format regex
Regex (?m-ix)^(?{\n\s{1,}"thread".*)

I want to read whole thread-1 is one message and other one is another message.

@simonasr
Copy link

simonasr commented Aug 4, 2020

This configuration works for us:

custom_parser.conf

[PARSER]
    Name multilinejson
    Format regex
    Regex ^(?<data>{.*)

Input tail config:

[INPUT]
    Name tail
    Multiline On
    Parser_Firstline multilinejson
    Tag testmultilinejson

Not sure if this is a best solution 😸
It would be great, if json parser could parse multiline json logs out of the box 🙏

@yeruvass
Copy link
Author

yeruvass commented Aug 4, 2020

Working perfectly. Thanks alot.

@bartweber
Copy link

Thanks @simonasr this is where I was looking for, for a long time. Though it looks like a workaround to me, I'm glad the issue didn't end in a show stopper. To get a regular JSON log event (without wrapping all content in the field data) I added an extra filter to your solution:

[FILTER] 
    Name             parser
    Key_Name         data
    Parser           json
    Unescape_Key     On

@github-actions
Copy link
Contributor

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions
Copy link
Contributor

github-actions bot commented May 5, 2021

This issue was closed because it has been stalled for 5 days with no activity.

@github-actions github-actions bot closed this as completed May 5, 2021
@edsiper
Copy link
Member

edsiper commented Jul 20, 2021

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants