Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiline functionality issue was solved by enabling docker_mode_parser, but after enabling the option i see kubernetes labels are missing from log output #2359

Closed
sateesh3550 opened this issue Jul 15, 2020 · 5 comments
Closed

Multiline functionality issue was solved by enabling docker_mode_parser, but after enabling the option i see kubernetes labels are missing from log output #2359

sateesh3550 opened this issue Jul 15, 2020 · 5 comments
Labels
Stale

Comments

@sateesh3550
Copy link

sateesh3550 commented Jul 15, 2020 •
edited
Loading

Bug Report

Describe the bug
<--- Multiline functionality issue was solved by enabling docker_mode_parser, but after enabling the option i see kubernetes labels are missing from log output. I was using the image 1.3.1 where i have having issues with multiline functionality and able to solve it by upgrading the image to 1.5.0 by enabling docker_mode_parser. But, after upgrading I see its missing kuberenetes labels -->

Log Sample

  • Example log message when i was using 1.3.1 image without having docker_mode_parser:
json:
level:{"log":"Error:
log:Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31\u0026args[]=object%20with%20keys%20%7Bname%2C%20slug%7D\u0026args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.\n","stream":"stderr","time":"2020-07-14T16:51:36.576674716Z"}
{"log":"    at T (/usr/src/app/.next/serverless/pages/index.js:62091:94)\n","stream":"stderr","time":"2020-07-14T16:51:36.576694028Z"}
{"log":"    at V (/usr/src/app/.next/serverless/pages/index.js:62091:229)\n","stream":"stderr","time":"2020-07-14T16:51:36.576697177Z"}
{"log":"    at X (/usr/src/app/.next/serverless/pages/index.js:62092:321)\n","stream":"stderr","time":"2020-07-14T16:51:36.576699823Z"}
{"log":"    at toArray (/usr/src/app/.next/serverless/pages/index.js:62093:398)\n","stream":"stderr","time":"2020-07-14T16:51:36.576704897Z"}
{"log":"    at a.b.renderDOM (/usr/src/app/.next/serverless/pages/index.js:17841:460)\n","stream":"stderr","time":"2020-07-14T16:51:36.576708829Z"}
{"log":"    at a.b.render (/usr/src/app/.next/serverless/pages/index.js:17834:326)\n","stream":"stderr","time":"2020-07-14T16:51:36.576712601Z"}
{"log":"    at a.b.read (/usr/src/app/.next/serverless/pages/index.js:17833:18)\n","stream":"stderr","time":"2020-07-14T16:51:36.576716237Z"}
{"log":"    at renderToString (/usr/src/app/.next/serverless/pages/index.js:17843:364)\n","stream":"stderr","time":"2020-07-14T16:51:36.576719756Z"}
{"log":"    at render (/usr/src/app/.next/serverless/pages/index.js:2487:16)\n","stream":"stderr","time":"2020-07-14T16:51:36.576723698Z"}
{"log":"    at Object.renderPage (/usr/src/app/.next/serverless/pages/index.js:2809:16)\n","stream":"stderr","time":"2020-07-14T16:51:36.576727194Z"}
timestamp:2020-07-14T16:51:36.576881Z
kubernetes:{ container_name: "test", docker_id: "docker123", pod_name: "test-7b8cb8df7d-mlsgj", namespace_name: "test" }

After enabling docker_mode_parser and docker_mode with image 1.5.0 i see missing kubernetes labels.

json:
log:Error: Minified React error #31;invariant=31&args[]=object%20with%20keys%20%7Bname%2C%20slug%7D&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at T (/usr/src/app/.next/serverless/pages/index.js:62091:94)
    at V (/usr/src/app/.next/serverless/pages/index.js:62091:229)
    at X (/usr/src/app/.next/serverless/pages/index.js:62092:321)
    at toArray (/usr/src/app/.next/serverless/pages/index.js:62093:398)
    at a.b.renderDOM (/usr/src/app/.next/serverless/pages/index.js:17841:460)
    at a.b.render (/usr/src/app/.next/serverless/pages/index.js:17834:326)
    at a.b.read (/usr/src/app/.next/serverless/pages/index.js:17833:18)
    at renderToString (/usr/src/app/.next/serverless/pages/index.js:17843:364)
    at render (/usr/src/app/.next/serverless/pages/index.js:2487:16)
    at Object.renderPage (/usr/src/app/.next/serverless/pages/index.js:2809:16)
stream:stderr
time:2020-07-15T14:20:56.240038601Z
timestamp:2020-07-15T14:21:05.331995Z

Here is the configuration i'm using. After changing the image to 1.5.0, i'm removing Multiline and adding Docker_Mode_Parser option to below configuartion

[INPUT]
Name tail
Tag test
Path /var/log/containers/test.log
Multiline_Flush 5
Multiline On
Parser_Firstline multi_line_logs
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10

[FILTER]
Name kubernetes
Match *
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_Tag_Prefix kube.var.log.containers.
Annotations On
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On

[PARSER]
Name multi_line_logs
Format regex
Regex ^(?[^\s]+:) (?.+)$
@ryjogo
Copy link

ryjogo commented Aug 10, 2020 •
edited
Loading

I am also seeing this with the below configuration:

[PARSER]
Name multi_line
Format regex
Regex (?<log>^{"log":"(\d{4}-\d{2}-\d{2}.*)|^{"log":"([\[].*|\w.*))

[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Refresh_Interval 5
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Docker_Mode On
Docker_Mode_Parser multi_line



[FILTER]
Name kubernetes
Match kube.*
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On

Using 1.5.2

@jullianoacqio
Copy link

For me it works in version 1.5.7, but the regular expression does not meet the expectations for my scenario.

Below is an excerpt from my sample log file.

{"log":"{ \"time\": \"2020-10-22 14:01:19,479\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:19] 'GET / HTTP/1.1' 200 -\", \"perfInfo\": \"562 7.820 6.613\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829354419968\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"200\", \"module\": \"_internal\" }\n","stream":"stderr","time":"2020-10-22T14:01:19.479252038Z"}
{"log":"{ \"time\": \"2020-10-22 14:01:20,353\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:20] 'GET / HTTP/1.1' 500 -\", \"perfInfo\": \"114 3.104 2.704\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"500\", \"module\": \"_internal\" }\n","stream":"stderr","time":"2020-10-22T14:01:20.353732743Z"}
{"log":"{ \"time\": \"2020-10-22 14:01:20,446\", \"process\": \"1\", \"severity\": \"ERROR\", \"name\": \"werkzeug\", \"message\": \"Error on request:\n","stream":"stderr","time":"2020-10-22T14:01:20.446809463Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2020-10-22T14:01:20.446860563Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/werkzeug/serving.py\", line 304, in run_wsgi\n","stream":"stderr","time":"2020-10-22T14:01:20.446869363Z"}
{"log":"    execute(self.server.app)\n","stream":"stderr","time":"2020-10-22T14:01:20.446896863Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/werkzeug/serving.py\", line 292, in execute\n","stream":"stderr","time":"2020-10-22T14:01:20.446902463Z"}
{"log":"    application_iter = app(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446906863Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/service/server.py\", line 439, in app\n","stream":"stderr","time":"2020-10-22T14:01:20.446915463Z"}
{"log":"    return self.app(e, s)\n","stream":"stderr","time":"2020-10-22T14:01:20.446919563Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py\", line 140, in application\n","stream":"stderr","time":"2020-10-22T14:01:20.446924063Z"}
{"log":"    return ProxyFix(application_unproxied)(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446928263Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py\", line 232, in __call__\n","stream":"stderr","time":"2020-10-22T14:01:20.446932463Z"}
{"log":"    return self.app(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446936764Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py\", line 117, in application_unproxied\n","stream":"stderr","time":"2020-10-22T14:01:20.446940564Z"}
{"log":"    result = odoo.http.root(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446944664Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1281, in __call__\n","stream":"stderr","time":"2020-10-22T14:01:20.446948364Z"}
{"log":"    return self.dispatch(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446952064Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1251, in __call__\n","stream":"stderr","time":"2020-10-22T14:01:20.446955964Z"}
{"log":"    return self.app(environ, start_wrapped)\n","stream":"stderr","time":"2020-10-22T14:01:20.446960264Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/werkzeug/middleware/shared_data.py\", line 220, in __call__\n","stream":"stderr","time":"2020-10-22T14:01:20.446964864Z"}
{"log":"    return self.app(environ, start_response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446969464Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1450, in dispatch\n","stream":"stderr","time":"2020-10-22T14:01:20.446973864Z"}
{"log":"    result = ir_http._dispatch()\n","stream":"stderr","time":"2020-10-22T14:01:20.446978764Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website_sale/models/ir_http.py\", line 15, in _dispatch\n","stream":"stderr","time":"2020-10-22T14:01:20.446983164Z"}
{"log":"    return super(IrHttp, cls)._dispatch()\n","stream":"stderr","time":"2020-10-22T14:01:20.446987064Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/ir_http.py\", line 175, in _dispatch\n","stream":"stderr","time":"2020-10-22T14:01:20.446991064Z"}
{"log":"    cls._register_website_track(response)\n","stream":"stderr","time":"2020-10-22T14:01:20.446995164Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/ir_http.py\", line 149, in _register_website_track\n","stream":"stderr","time":"2020-10-22T14:01:20.446998764Z"}
{"log":"    request.env['website.visitor']._handle_webpage_dispatch(response, website_page)\n","stream":"stderr","time":"2020-10-22T14:01:20.447003164Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 195, in _handle_webpage_dispatch\n","stream":"stderr","time":"2020-10-22T14:01:20.447008764Z"}
{"log":"    self._handle_website_page_visit(response, website_page, visitor_sudo)\n","stream":"stderr","time":"2020-10-22T14:01:20.447013064Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 212, in _handle_website_page_visit\n","stream":"stderr","time":"2020-10-22T14:01:20.447017464Z"}
{"log":"    visitor_sudo._add_tracking(domain, website_track_values)\n","stream":"stderr","time":"2020-10-22T14:01:20.447027464Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 222, in _add_tracking\n","stream":"stderr","time":"2020-10-22T14:01:20.447031864Z"}
{"log":"    self.env['website.track'].create(website_track_values)\n","stream":"stderr","time":"2020-10-22T14:01:20.447036264Z"}
{"log":"  File \"\u003cdecorator-gen-3\u003e\", line 2, in create\n","stream":"stderr","time":"2020-10-22T14:01:20.447040464Z"}
{"log":"    \n","stream":"stderr","time":"2020-10-22T14:01:20.447045364Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/api.py\", line 343, in _model_create_multi\n","stream":"stderr","time":"2020-10-22T14:01:20.447049564Z"}
{"log":"    return create(self, [arg])\n","stream":"stderr","time":"2020-10-22T14:01:20.447054164Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/models.py\", line 3732, in create\n","stream":"stderr","time":"2020-10-22T14:01:20.447058564Z"}
{"log":"    records = self._create(data_list)\n","stream":"stderr","time":"2020-10-22T14:01:20.447063064Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/models.py\", line 3818, in _create\n","stream":"stderr","time":"2020-10-22T14:01:20.447066864Z"}
{"log":"    cr.execute(query, params)\n","stream":"stderr","time":"2020-10-22T14:01:20.447071164Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/sql_db.py\", line 163, in wrapper\n","stream":"stderr","time":"2020-10-22T14:01:20.447075364Z"}
{"log":"    return f(self, *args, **kwargs)\n","stream":"stderr","time":"2020-10-22T14:01:20.447080164Z"}
{"log":"  File \"/usr/lib/python3/dist-packages/odoo/sql_db.py\", line 240, in execute\n","stream":"stderr","time":"2020-10-22T14:01:20.447084264Z"}
{"log":"    res = self._obj.execute(query, params)\n","stream":"stderr","time":"2020-10-22T14:01:20.447088964Z"}
{"log":"psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update\n","stream":"stderr","time":"2020-10-22T14:01:20.447093364Z"}
{"log":"CONTEXT:  SQL statement \"SELECT 1 FROM ONLY \"public\".\"website_visitor\" x WHERE \"id\" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x\"\", \"perfInfo\": \"- - -\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"t:\n","stream":"stderr","time":"2020-10-22T14:01:20.447098364Z"}
{"log":"{ \"time\": \"2020-10-22 14:01:19,479\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:19] 'GET / HTTP/1.1' 200 -\", \"perfInfo\": \"562 7.820 6.613\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829354419968\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"200\", \"module\": \"_internal\" }\n","stream":"stderr","time":"2020-10-22T14:01:19.479252038Z"}
{"log":"{ \"time\": \"2020-10-22 14:01:20,353\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:20] 'GET / HTTP/1.1' 500 -\", \"perfInfo\": \"114 3.104 2.704\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"500\", \"module\": \"_internal\" }\n","stream":"stderr","time":"2020-10-22T14:01:20.353732743Z"}
{"log":"{ \"time\": \"2020-10-22 14:01:20,353\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:20] 'GET / HTTP/1.1' 500 -\", \"perfInfo\": \"114 3.104 2.704\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"500\", \"module\": \"_internal\" }\n","stream":"stderr","time":"2020-10-22T14:01:20.353732743Z"}

My result is:

{"@timestamp":"2020-10-22T14:01:20.446809Z","log":"{ \"time\": \"2020-10-22 14:01:19,479\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:19] 'GET / HTTP/1.1' 200 -\", \"perfInfo\": \"562 7.820 6.613\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829354419968\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"200\", \"module\": \"_internal\" }\n{ \"time\": \"2020-10-22 14:01:20,353\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:20] 'GET / HTTP/1.1' 500 -\", \"perfInfo\": \"114 3.104 2.704\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"500\", \"module\": \"_internal\" }\n{ \"time\": \"2020-10-22 14:01:20,446\", \"process\": \"1\", \"severity\": \"ERROR\", \"name\": \"werkzeug\", \"message\": \"Error on request:\n","stream":"stderr","docker_time":"2020-10-22T14:01:20.446809463Z","collector_app":"fluentbit"}
{"@timestamp":"2020-10-22T14:01:20.447088Z","log":"Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/werkzeug/serving.py\", line 304, in run_wsgi\n    execute(self.server.app)\n  File \"/usr/lib/python3/dist-packages/werkzeug/serving.py\", line 292, in execute\n    application_iter = app(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/odoo/service/server.py\", line 439, in app\n    return self.app(e, s)\n  File \"/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py\", line 140, in application\n    return ProxyFix(application_unproxied)(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py\", line 232, in __call__\n    return self.app(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py\", line 117, in application_unproxied\n    result = odoo.http.root(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1281, in __call__\n    return self.dispatch(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1251, in __call__\n    return self.app(environ, start_wrapped)\n  File \"/usr/lib/python3/dist-packages/werkzeug/middleware/shared_data.py\", line 220, in __call__\n    return self.app(environ, start_response)\n  File \"/usr/lib/python3/dist-packages/odoo/http.py\", line 1450, in dispatch\n    result = ir_http._dispatch()\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website_sale/models/ir_http.py\", line 15, in _dispatch\n    return super(IrHttp, cls)._dispatch()\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/ir_http.py\", line 175, in _dispatch\n    cls._register_website_track(response)\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/ir_http.py\", line 149, in _register_website_track\n    request.env['website.visitor']._handle_webpage_dispatch(response, website_page)\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 195, in _handle_webpage_dispatch\n    self._handle_website_page_visit(response, website_page, visitor_sudo)\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 212, in _handle_website_page_visit\n    visitor_sudo._add_tracking(domain, website_track_values)\n  File \"/usr/lib/python3/dist-packages/odoo/addons/website/models/website_visitor.py\", line 222, in _add_tracking\n    self.env['website.track'].create(website_track_values)\n  File \"<decorator-gen-3>\", line 2, in create\n    \n  File \"/usr/lib/python3/dist-packages/odoo/api.py\", line 343, in _model_create_multi\n    return create(self, [arg])\n  File \"/usr/lib/python3/dist-packages/odoo/models.py\", line 3732, in create\n    records = self._create(data_list)\n  File \"/usr/lib/python3/dist-packages/odoo/models.py\", line 3818, in _create\n    cr.execute(query, params)\n  File \"/usr/lib/python3/dist-packages/odoo/sql_db.py\", line 163, in wrapper\n    return f(self, *args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/odoo/sql_db.py\", line 240, in execute\n    res = self._obj.execute(query, params)\n","stream":"stderr","docker_time":"2020-10-22T14:01:20.447088964Z","collector_app":"fluentbit"}
{"@timestamp":"2020-10-22T14:01:20.447093Z","log":"psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update\n","stream":"stderr","docker_time":"2020-10-22T14:01:20.447093364Z","collector_app":"fluentbit"}
{"@timestamp":"2020-10-22T14:01:20.353732Z","log":"CONTEXT:  SQL statement \"SELECT 1 FROM ONLY \"public\".\"website_visitor\" x WHERE \"id\" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x\"\", \"perfInfo\": \"- - -\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"t:\n{ \"time\": \"2020-10-22 14:01:19,479\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:19] 'GET / HTTP/1.1' 200 -\", \"perfInfo\": \"562 7.820 6.613\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829354419968\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"200\", \"module\": \"_internal\" }\n{ \"time\": \"2020-10-22 14:01:20,353\", \"process\": \"1\", \"severity\": \"INFO\", \"name\": \"werkzeug\", \"message\": \"177.159.211.76 - - [22/Oct/2020 14:01:20] 'GET / HTTP/1.1' 500 -\", \"perfInfo\": \"114 3.104 2.704\", \"processName\": \"MainProcess\", \"threadName\": \"odoo.service.http.request.139829346027264\", \"pathname\": \"/usr/lib/python3/dist-packages/werkzeug/_internal.py\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"lineno\": \"122\", \"metric\": { }, \"httpStatus\": \"500\", \"module\": \"_internal\" }\n","stream":"stderr","docker_time":"2020-10-22T14:01:20.353732743Z","collector_app":"fluentbit"}

My custom_parsers.conf:

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name    k8s-custom-tag
        Format  regex
        Regex   (?<namespace_name>[^_.]+)\.(?<container_name>[^.]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*+)\.(?<docker_id>[a-z0-9]{64})$

    [PARSER]
        Name         multi_line
        Format       regex
        # Regex        (\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})
        Regex        (?<log>^{"log":"(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})|^{"log":"([\[].*|\w.*))

My fluent-bit.conf:

    [SERVICE]
        Flush                     1
        Log_Level                 info
        Daemon                    off
        Parsers_File              custom_parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        storage.path              /var/log/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M

    [INPUT]
        Name              tail
        Tag_Regex         (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
        Tag               k8s.<namespace_name>.<container_name>.<pod_name>.<docker_id>
        Parser            docker
        Path              /fluent-bit/etc/*.log
        # Path              /var/log/containers/*.log
        Exclude_Path      *_kube-system_*.log, *_kube-node-lease_*.log, *_kube-public_*.log, *_fluentbit-system_*.log
        Mem_Buf_Limit     10M
        Ignore_Older      2d
        Skip_Long_Lines   On
        Refresh_Interval  10
        Docker_Mode       On
        Docker_Mode_Parser multi_line
        DB                /var/log/flb_kube.db
        DB.Sync           normal

    [FILTER]
        Name                        kubernetes
        Match                       k8s.*
        Kube_Tag_Prefix             k8s.
        Regex_Parser                k8s-custom-tag
        Merge_Log                   On
        Merge_Log_Key               logprocessed
        Merge_Log_Trim              On
        Keep_Log                    Off
        K8S-Logging.Exclude         On
        Kube_meta_preload_cache_dir On
        DNS_Retries                 6
        DNS_Wait_Time               30

    [FILTER]
        Name    modify
        Match   k8s.*
        Add     collector_app   fluentbit
        Rename  time            docker_time

    [OUTPUT]
        Name             stdout
        Match            k8s.*
        Format           json_lines
        json_date_key    @timestamp
        json_date_format iso8601

@edsiper edsiper mentioned this issue May 3, 2021
Merged
@github-actions
Copy link
Contributor

github-actions bot commented May 6, 2021

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the Stale label May 6, 2021
@github-actions
Copy link
Contributor

This issue was closed because it has been stalled for 5 days with no activity.

@edsiper
Copy link
Member

edsiper commented Jul 20, 2021

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@edsiper @ryjogo @sateesh3550 @jullianoacqio