parsers: Support CRI-O and containerd #881

StevenACoffman · 2018-11-02T22:12:57Z

See #876 and #873
If a single character is detected, consider this the log tag for the line.
This is a part of the multiline handling for cri-o logs.

Example Fluent-bit log line on disk:

root@fluent-bit-29xjh:/# more /var/log/containers/fluent-bit-29xjh_dev-eeva-logging_fluent-bit-0db128860717a93c7d219425c9d49f4445c2d6c6e5fed3f4dc4298f03c582378.log 
2018-10-31T21:54:45.45487617Z stderr F Fluent-Bit v0.14.4
2018-10-31T21:54:45.454938595Z stderr F Copyright (C) Treasure Data
2018-10-31T21:54:45.454946545Z stderr F 
2018-10-31T21:54:45.455512949Z stderr F [2018/10/31 21:54:45] [ info] [engine] started (pid=1)
2018-10-31T21:54:48.837184585Z stderr F [2018/10/31 21:54:48] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
2018-10-31T21:54:48.837218858Z stderr F [2018/10/31 21:54:48] [ info] [filter_kube] local POD info OK
2018-10-31T21:54:48.837226848Z stderr F [2018/10/31 21:54:48] [ info] [filter_kube] testing connectivity with API server...
2018-10-31T21:54:49.036009069Z stderr F [2018/10/31 21:54:49] [ info] [filter_kube] API server connectivity OK
2018-10-31T21:54:49.036783868Z stderr F [2018/10/31 21:54:49] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

Example Elasticsearch JSON document:

{
  "_index": "logstash-2018.11.03",
  "_type": "flb_type",
  "_id": "sdIP22YBrbPWNMRHKBwz",
  "_version": 1,
  "_score": null,
  "_source": {
    "time": "2018-11-03T19:31:10.000Z",
    "stream": "stderr",
    "logtag": "F",
    "message": "\u001b[1mFluent-Bit v0.14.6\u001b[0m",
    "kubernetes": {
      "pod_name": "fluent-bit-wjnfq",
      "namespace_name": "dev-logging",
      "pod_id": "0293ad2d-df9f-11e8-8914-2e491e409191",
      "labels": {
        "controller-revision-hash": "1569803976",
        "k8s-app": "fluent-bit-logging",
        "kubernetes_io/cluster-service": "true",
        "pod-template-generation": "1",
        "version": "v1"
      },
      "annotations": {
        "prometheus_io/path": "/api/v1/metrics/prometheus",
        "prometheus_io/port": "2020",
        "prometheus_io/scrape": "true"
      },
      "host": "10.63.56.245",
      "container_name": "fluent-bit",
      "docker_id": "013362a984722805ede9cf323e34919890ea44ea0004897d3ed93178116676a2"
    }
  },
  "fields": {
    "time": [
      "2018-11-03T19:31:10.000Z"
    ]
  },
  "sort": [
    1541273470000
  ]
}

See fluent#876 and fluent#873 Signed-off-by: Steve Coffman <[email protected]>

edsiper · 2018-11-02T23:13:05Z

@StevenACoffman are we ok to merge this one ?

StevenACoffman · 2018-11-03T11:37:57Z

@kskewes If you get a chance to run this through ElasticSearch, I would appreciate a comment with the output for documentation purposes. I have tested it with Kafka output and it works fine, but don't have an ElasticSearch cluster setup.

karlskewes · 2018-11-03T19:02:58Z

conf/parsers.conf

    Format Regex
-    Regex /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
+    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%N%:z


Is this okay? %N and %:z

Should it be this?
%Y-%m-%dT%H:%M:%S.%L%z

The colon appears wrong. What is %L? Is it milliseconds as a decimal number [000, 999]? I don't see %L in the strptime documentation. I think I got %N from unix date man page.

The timestamps I see follow this format:

2016-02-17T00:04:05.931087621Z

With both your suggestions I see them in Kafka fine. Perhaps that's why I needed the Time_Keep On before?

Indeed wrt %L.
It's not in that documentation but appears to be a custom fluent bit option for nano seconds.
See bottom of this page: https://fluentbit.io/documentation/0.14/parser/

karlskewes · 2018-11-03T19:03:57Z

conf/parsers.conf

-    # http://rubular.com/r/izM6olvshn
-    Name crio
+    # http://rubular.com/r/tjUt3Awgg4
+    Name cri
    Format Regex


We could make this regex although there was a commit to make this case insensitive merged.

karlskewes · 2018-11-03T19:29:34Z

conf/parsers.conf

    Format Regex
-    Regex /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
+    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%N%:z
    Time_Keep   On


Time_Keep On seems to break Elasticsearch for me. Could be my K8s filter or ES output?
Fluent-Bit logs this error.

[2018/11/03 19:26:13] [ warn] [out_es] Elasticsearch error {"took":150,"errors":true,"items":[{"index":{"_index":"logstash-2018.11.03","_type":"flb_type","_id":"_vEK22YBtlziBPpiPhwA","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"i_o_exception","reason":"Duplicate field 'time'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@2d5483; line: 1, column: 43]"}}}},{"index":{"_index":"logstash-2018.11.03","_type":"flb_type","_id":"__EK22YBtlziBPpiPhwA","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"i_o_exception","reason":"Duplicate field 'time'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@4988de6b; line: 1, column: 43]"}}}},{"index":{"_index":"logstash-2018.11.03","_type":"flb_type","_id":"APEK22YBtlziBPpiPh0A","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_b

Doesn't make a difference for my use case, so I can happily remove that

Thanks. Do you end up with additional time fields in Kafka?
I can also remove locally and wonder if one of the maintainers or someone with more experience with this field can advise. I haven't been able to grok the use case.

karlskewes · 2018-11-03T19:34:09Z

With above changes I get this in Elasticsearch:

{
  "_index": "logstash-2018.11.03",
  "_type": "flb_type",
  "_id": "sdIP22YBrbPWNMRHKBwz",
  "_version": 1,
  "_score": null,
  "_source": {
    "time": "2018-11-03T19:31:10.000Z",
    "stream": "stderr",
    "logtag": "F",
    "message": "\u001b[1mFluent-Bit v0.14.6\u001b[0m",
    "kubernetes": {
      "pod_name": "fluent-bit-wjnfq",
      "namespace_name": "dev-logging",
      "pod_id": "0293ad2d-df9f-11e8-8914-2e491e409191",
      "labels": {
        "controller-revision-hash": "1569803976",
        "k8s-app": "fluent-bit-logging",
        "kubernetes_io/cluster-service": "true",
        "pod-template-generation": "1",
        "version": "v1"
      },
      "annotations": {
        "prometheus_io/path": "/api/v1/metrics/prometheus",
        "prometheus_io/port": "2020",
        "prometheus_io/scrape": "true"
      },
      "host": "10.63.56.245",
      "container_name": "fluent-bit",
      "docker_id": "013362a984722805ede9cf323e34919890ea44ea0004897d3ed93178116676a2"
    }
  },
  "fields": {
    "time": [
      "2018-11-03T19:31:10.000Z"
    ]
  },
  "sort": [
    1541273470000
  ]
}

Signed-off-by: Steve Coffman <[email protected]>

StevenACoffman · 2018-11-04T17:43:52Z

Argh. My parser configuration is applied via a configmap, rather than what is baked into the docker image, so I tried copy pasting what I was running in my cluster using my phone yesterday. That was ill-advised, and I'm sorry for the confusion. This corrects the mistake. Thank you @kskewes

karlskewes · 2018-11-04T17:56:56Z

That would be tricky!
No problem. Was a good opportunity to revisit the docs and double check what we're doing here. Thanks~

StevenACoffman · 2018-11-05T10:19:29Z

@edsiper This is okay to merge now.

edsiper · 2018-11-06T17:16:45Z

thanks!

* pipelines: inputs: exec_wasi: Add document for in_exec_wasi plugin Signed-off-by: Hiroshi Hatake <[email protected]> * summary: Add Exec Wasi section Signed-off-by: Hiroshi Hatake <[email protected]> * pipelines: filters: wasm: Add document for filter_wasm plugin Signed-off-by: Hiroshi Hatake <[email protected]> * summary: Add wasm section Signed-off-by: Hiroshi Hatake <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]>

Support CRI-O and containerd

6471706

See fluent#876 and fluent#873 Signed-off-by: Steve Coffman <[email protected]>

StevenACoffman force-pushed the master branch from 9019194 to 6471706 Compare November 2, 2018 22:16

This was referenced Nov 2, 2018

parsers: Add containerd log parser #873

Closed

Add logtag field to crio logs #876

Closed

edsiper self-assigned this Nov 2, 2018

edsiper added enhancement waiting-for-user Waiting for more information, tests or requested changes labels Nov 2, 2018

karlskewes reviewed Nov 3, 2018

View reviewed changes

StevenACoffman force-pushed the master branch from 63ef172 to cf53fba Compare November 4, 2018 02:09

Fix based on @kskewes from fluent#881

291f836

Signed-off-by: Steve Coffman <[email protected]>

StevenACoffman force-pushed the master branch from cf53fba to 291f836 Compare November 4, 2018 17:42

edsiper merged commit 767cc53 into fluent:master Nov 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parsers: Support CRI-O and containerd #881

parsers: Support CRI-O and containerd #881

StevenACoffman commented Nov 2, 2018 •

edited

Loading

edsiper commented Nov 2, 2018

StevenACoffman commented Nov 3, 2018 •

edited

Loading

karlskewes Nov 3, 2018 •

edited

Loading

StevenACoffman Nov 3, 2018 •

edited

Loading

karlskewes Nov 4, 2018

karlskewes Nov 3, 2018

karlskewes Nov 3, 2018 •

edited

Loading

StevenACoffman Nov 3, 2018

karlskewes Nov 3, 2018

karlskewes commented Nov 3, 2018

StevenACoffman commented Nov 4, 2018 •

edited

Loading

karlskewes commented Nov 4, 2018

StevenACoffman commented Nov 5, 2018

edsiper commented Nov 6, 2018

parsers: Support CRI-O and containerd #881

parsers: Support CRI-O and containerd #881

Conversation

StevenACoffman commented Nov 2, 2018 • edited Loading

edsiper commented Nov 2, 2018

StevenACoffman commented Nov 3, 2018 • edited Loading

karlskewes Nov 3, 2018 • edited Loading

Choose a reason for hiding this comment

StevenACoffman Nov 3, 2018 • edited Loading

Choose a reason for hiding this comment

karlskewes Nov 4, 2018

Choose a reason for hiding this comment

karlskewes Nov 3, 2018

Choose a reason for hiding this comment

karlskewes Nov 3, 2018 • edited Loading

Choose a reason for hiding this comment

StevenACoffman Nov 3, 2018

Choose a reason for hiding this comment

karlskewes Nov 3, 2018

Choose a reason for hiding this comment

karlskewes commented Nov 3, 2018

StevenACoffman commented Nov 4, 2018 • edited Loading

karlskewes commented Nov 4, 2018

StevenACoffman commented Nov 5, 2018

edsiper commented Nov 6, 2018

StevenACoffman commented Nov 2, 2018 •

edited

Loading

StevenACoffman commented Nov 3, 2018 •

edited

Loading

karlskewes Nov 3, 2018 •

edited

Loading

StevenACoffman Nov 3, 2018 •

edited

Loading

karlskewes Nov 3, 2018 •

edited

Loading

StevenACoffman commented Nov 4, 2018 •

edited

Loading