Fluent Bit restarts if a json message has large number of keys

[ujjwalgarg1995]

Bug Report

Describe the bug
Fluent Bit restarts if a json message has large number of keys.

Message: {"message":"DDP connection failed.","timestamp":"2021-08-03T13:33:23.413Z","level":"error","context":{"0":"N","1":"e","2":"t","3":"w","4":"o","5":"r","6":"k","7":" ","8":"e","9":"r","10":"r","11":"o","12":"r","13":":","14":" ","15":"w","16":"s","17":"s","18":":","19":"/","20":"/","21":"d","22":"e","23":"v","24":"s","25":"t","26":"a","27":"b","28":"l","29":"e","30":".","31":"e","32":"d","33":"u","34":".","35":"b","36":"a","37":"r","38":"c","39":"o","40":".","41":"c","42":"l","43":"o","44":"u","45":"d","46":":","47":"4","48":"4","49":"3","50":"/","51":"w","52":"e","53":"b","54":"s","55":"o","56":"c","57":"k","58":"e","59":"t","60":":","61":" ","62":"H","63":"o","64":"s","65":"t","66":"n","67":"a","68":"m","69":"e","70":"/","71":"I","72":"P","73":" ","74":"d","75":"o","76":"e","77":"s","78":" ","79":"n","80":"o","81":"t","82":" ","83":"m","84":"a","85":"t","86":"c","87":"h","88":" ","89":"c","90":"e","91":"r","92":"t","93":"i","94":"f","95":"i","96":"c","97":"a","98":"t","99":"e","100":"'","101":"s","102":" ","103":"a","104":"l","105":"t","106":"n","107":"a","108":"m","109":"e","110":"s","111":":","112":" ","113":"H","114":"o","115":"s","116":"t","117":":","118":" ","119":"d","120":"e","121":"v","122":"s","123":"t","124":"a","125":"b","126":"l","127":"e","128":".","129":"e","130":"d","131":"u","132":".","133":"b","134":"a","135":"r","136":"c","137":"o","138":".","139":"c","140":"l","141":"o","142":"u","143":"d","144":".","145":" ","146":"i","147":"s","148":" ","149":"n","150":"o","151":"t","152":" ","153":"i","154":"n","155":" ","156":"t","157":"h","158":"e","159":" ","160":"c","161":"e","162":"r","163":"t","164":"'","165":"s","166":" ","167":"a","168":"l","169":"t","170":"n","171":"a","172":"m","173":"e","174":"s","175":":","176":" ","177":"D","178":"N","179":"S","180":":","181":"e","182":"d","183":"u","184":".","185":"b","186":"a","187":"r","188":"c","189":"o","190":".","191":"c","192":"o","193":"m","194":",","195":" ","196":"D","197":"N","198":"S","199":":","200":"","201":".","202":"e","203":"d","204":"u","205":".","206":"b","207":"a","208":"r","209":"c","210":"o","211":".","212":"c","213":"o","214":"m","215":",","216":" ","217":"D","218":"N","219":"S","220":":","221":"","222":".","223":"c","224":"l","225":"i","226":"c","227":"k","228":"s","229":"h","230":"a","231":"r","232":"e","233":".","234":"a","235":"p","236":"p","237":",","238":" ","239":"D","240":"N","241":"S","242":":","243":"","244":".","245":"n","246":"e","247":"x","248":"x","249":"i","250":"s","251":".","252":"b","253":"a","254":"r","255":"c","256":"o","257":".","258":"c","259":"o","260":"m","261":",","262":" ","263":"D","264":"N","265":"S","266":":","267":"o","268":"v","269":"e","270":"r","271":"t","272":"u","273":"r","274":"e","275":".","276":"b","277":"a","278":"r","279":"c","280":"o","281":".","282":"c","283":"o","284":"m","285":",","286":" ","287":"D","288":"N","289":"S","290":":","291":"","292":".","293":"m","294":"e","295":"d","296":"i","297":"a","298":"l","299":"o","300":"n","301":".","302":"b","303":"a","304":"r","305":"c","306":"o","307":".","308":"c","309":"o","310":"m","311":",","312":" ","313":"D","314":"N","315":"S","316":":","317":"s","318":"t","319":"o","320":"r","321":"e","322":".","323":"b","324":"a","325":"r","326":"c","327":"o","328":".","329":"c","330":"o","331":"m","332":",","333":" ","334":"D","335":"N","336":"S","337":":","338":"","339":".","340":"p","341":"r","342":"o","343":"j","344":".","345":"b","346":"a","347":"r","348":"c","349":"o","350":".","351":"c","352":"o","353":"m","354":",","355":" ","356":"D","357":"N","358":"S","359":":","360":"u","361":"n","362":"i","363":"v","364":"e","365":"r","366":"s","367":"i","368":"t","369":"y","370":".","371":"b","372":"a","373":"r","374":"c","375":"o","376":".","377":"c","378":"o","379":"m","380":",","381":" ","382":"D","383":"N","384":"S","385":":","386":"c","387":"m","388":"p","389":".","390":"b","391":"a","392":"r","393":"c","394":"o","395":".","396":"c","397":"o","398":"m","399":",","400":" ","401":"D","402":"N","403":"S","404":":","405":"i","406":"n","407":"s","408":"i","409":"g","410":"h","411":"t","412":"s","413":".","414":"b","415":"a","416":"r","417":"c","418":"o","419":".","420":"c","421":"o","422":"m","423":",","424":" ","425":"D","426":"N","427":"S","428":":","429":"","430":".","431":"i","432":"n","433":"s","434":"i","435":"g","436":"h","437":"t","438":"s","439":".","440":"b","441":"a","442":"r","443":"c","444":"o","445":".","446":"c","447":"o","448":"m","449":",","450":" ","451":"D","452":"N","453":"S","454":":","455":"","456":".","457":"w","458":"s","459":"g","460":".","461":"b","462":"a","463":"r","464":"c","465":"o","466":".","467":"c","468":"o","469":"m","470":",","471":" ","472":"D","473":"N","474":"S","475":":","476":"m","477":"e","478":"d","479":"i","480":"a","481":"l","482":"o","483":"n","484":".","485":"b","486":"a","487":"r","488":"c","489":"o","490":".","491":"c","492":"o","493":"m","494":",","495":" ","496":"D","497":"N","498":"S","499":":","500":"n","501":"e","502":"x","503":"x","504":"i","505":"s","506":".","507":"b","508":"a","509":"r","510":"c","511":"o","512":".","513":"c","514":"o","515":"m","516":",","517":" ","518":"D","519":"N","520":"S","521":":","522":"a","523":"c","524":"t","525":"i","526":"v","527":"a","528":"t","529":"e","530":".","531":"b","532":"a","533":"r","534":"c","535":"o","536":".","537":"c","538":"o","539":"m","540":",","541":" ","542":"D","543":"N","544":"S","545":":","546":"","547":".","548":"c","549":"l","550":"o","551":"u","552":"d","553":".","554":"b","555":"a","556":"r","557":"c","558":"o","559":".","560":"c","561":"o","562":"m","563":",","564":" ","565":"D","566":"N","567":"S","568":":","569":"","570":".","571":"o","572":"v","573":"e","574":"r","575":"t","576":"u","577":"r","578":"e","579":".","580":"b","581":"a","582":"r","583":"c","584":"o","585":".","586":"c","587":"o","588":"m","589":",","590":" ","591":"D","592":"N","593":"S","594":":","595":"c","596":"l","597":"i","598":"c","599":"k","600":"s","601":"h","602":"a","603":"r","604":"e","605":".","606":"a","607":"p","608":"p","609":",","610":" ","611":"D","612":"N","613":"S","614":":","615":"","616":".","617":"u","618":"n","619":"i","620":"v","621":"e","622":"r","623":"s","624":"i","625":"t","626":"y","627":".","628":"b","629":"a","630":"r","631":"c","632":"o","633":".","634":"c","635":"o","636":"m","637":",","638":" ","639":"D","640":"N","641":"S","642":":","643":"","644":".","645":"s","646":"t","647":"o","648":"r","649":"e","650":".","651":"b","652":"a","653":"r","654":"c","655":"o","656":".","657":"c","658":"o","659":"m","660":",","661":" ","662":"D","663":"N","664":"S","665":":","666":"x","667":"m","668":"s","669":".","670":"b","671":"a","672":"r","673":"c","674":"o","675":".","676":"c","677":"o","678":"m","679":",","680":" ","681":"D","682":"N","683":"S","684":":","685":"","686":".","687":"a","688":"c","689":"t","690":"i","691":"v","692":"a","693":"t","694":"e","695":".","696":"b","697":"a","698":"r","699":"c","700":"o","701":".","702":"c","703":"o","704":"m","705":",","706":" ","707":"D","708":"N","709":"S","710":":","711":"w","712":"e","713":"c","714":"o","715":"n","716":"n","717":"e","718":"c","719":"t","720":".","721":"b","722":"a","723":"r","724":"c","725":"o","726":".","727":"c","728":"o","729":"m","730":",","731":" ","732":"D","733":"N","734":"S","735":":","736":"","737":".","738":"w","739":"e","740":"c","741":"o","742":"n","743":"n","744":"e","745":"c","746":"t","747":".","748":"b","749":"a","750":"r","751":"c","752":"o","753":".","754":"c","755":"o","756":"m","757":",","758":" ","759":"D","760":"N","761":"S","762":":","763":"","764":".","765":"c","766":"m","767":"p","768":".","769":"b","770":"a","771":"r","772":"c","773":"o","774":".","775":"c","776":"o","777":"m","778":",","779":" ","780":"D","781":"N","782":"S","783":":","784":"w","785":"s","786":"g","787":".","788":"b","789":"a","790":"r","791":"c","792":"o","793":".","794":"c","795":"o","796":"m","797":",","798":" ","799":"D","800":"N","801":"S","802":":","803":"p","804":"r","805":"o","806":"j","807":".","808":"b","809":"a","810":"r","811":"c","812":"o","813":".","814":"c","815":"o","816":"m","817":",","818":" ","819":"D","820":"N","821":"S","822":":","823":"c","824":"l","825":"o","826":"u","827":"d","828":".","829":"b","830":"a","831":"r","832":"c","833":"o","834":".","835":"c","836":"o","837":"m","838":",","839":" ","840":"D","841":"N","842":"S","843":":","844":"","845":".","846":"x","847":"m","848":"s","849":".","850":"b","851":"a","852":"r","853":"c","854":"o","855":".","856":"c","857":"o","858":"m","859":",","860":" ","861":"D","862":"N","863":"S","864":":","865":"","866":".","867":"b","868":"i","869":"n","870":".","871":"c","872":"l","873":"o","874":"u","875":"d","876":".","877":"b","878":"a","879":"r","880":"c","881":"o","882":".","883":"c","884":"o","885":"m","service":"ROOT","id":"vwhjvhvjs","serviceMetadata":{"meetingId":"","instituteId":"obyr","role":"abc"}}}

To Reproduce

• Add a input tail plugin that reads from a file
• Paste the above message in that file
• Check Fluent Bit Status

Aug 18 11:51:33 nrc-000babcd6ada barco-nrc-td-agent-bit[13826]: [2021/08/18 11:51:33] [debug] [storage] [cio file] alloc_size from 4096 to 36864
Aug 18 11:51:33 nrc-000babcd6ada barco-nrc-td-agent-bit[13826]: [2021/08/18 11:51:33] [debug] [filter:nest:nest.5] Lift : Outer map size is 5, will be 893, lifting 1 record(s)
Aug 18 11:51:33 nrc-000babcd6ada barco-nrc-td-agent-bit[13826]: [2021/08/18 11:51:33] [debug] [filter:nest:nest.6] Lift : Outer map size is 893, will be 895, lifting 1 record(s)
Aug 18 11:51:33 nrc-000babcd6ada barco-nrc-td-agent-bit[13826]: [2021/08/18 11:51:33] [engine] caught signal (SIGSEGV)
Aug 18 11:51:33 nrc-000babcd6ada barco-nrc-td-agent-bit[13826]: #0  0x562822076522      in  cb_modifier_filter() at plugins/filter_record_modifier/filter_modifier.c:307
Aug 18 11:51:33 nrc-000babcd6ada systemd[1]: barco-nrc-td-agent-bit.service: Main process exited, code=killed, status=11/SEGV
Aug 18 11:51:33 nrc-000babcd6ada systemd[1]: barco-nrc-td-agent-bit.service: Failed with result 'signal'.

• Steps to reproduce the problem:

Expected behavior
Fluent Bit should gracefully handle when using "nest" filter and "Nested_under" key has large number of keys

Screenshots

Your Environment

• Version used: 1.8.1, 1.8.3
• Configuration:
• [INPUT]
  name tail
  storage.type filesystem
  Mem_Buf_Limit 5M
  Buffer_Chunk_Size 32KB
  Buffer_Max_Size 1M
  Path /var/log/abc/def*.log
  DB /etc/barco-nrc-td-agent-bit/log-abc.db
  Skip_Long_Lines on
  Path_Key file
  tag app-logs
  Parser app_parser

[FILTER]
Name nest
Match app-logs
Operation lift
Nested_under context

[FILTER]
Name nest
Match app-logs
Operation lift
Nested_under serviceMetadata

[FILTER]
Name record_modifier
Match app-logs
Record version ${VERSION}
Record env ${ENV}
Record component xyz

[FILTER]
Name nest
Match app-logs
Operation nest
Wildcard version
Wildcard namw
Nest_under serviceMetadata

[FILTER]
Name nest
Match app-logs
Operation nest
Wildcard serviceMetadata
Nest_under context

• Environment name and version (e.g. Kubernetes? What version?): Tail

• Server type and version:

• Operating System and version:

• Filters and plugins:

Additional context
I tried using "grep" plugin to discard any such message.
[FILTER]
Name grep
Match *
Exclude context ([^:]:){125}([^:])
Aim is to discard any message that has more than 125 keys but it is not working. I am not sure if regular expression in the Exclude statement is not working or my approach is incorrect.
Also, after doing some google search, I changed it to Exclude context ([^:]:){125}([^:])\ but even that didn't work.

What are the best practices to handle these scenarios?
Is there any limit on number of keys that fluent bit can handle?

[ujjwalgarg1995]

@edsiper can you please help me with this?

[nokute78]

I can reproduce.

1. fluent-bit -c a.conf
2. curl -d @json.log -XPOST -H "content-type: application/json" http://localhost:9880/json.log

json.log

a.conf:

[INPUT]
    Name http

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under context

[FILTER]
    Name record_modifier
    Match *

[OUTPUT]
    Name stdout

[nokute78]

I sent a patch #3982.