Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support additional parser configuration: ndjson and multiline in container logs data-stream #2345

Merged
merged 8 commits into from
Dec 17, 2021
Merged

Support additional parser configuration: ndjson and multiline in container logs data-stream #2345

merged 8 commits into from
Dec 17, 2021

Conversation

tetianakravchenko
Copy link
Contributor

@tetianakravchenko tetianakravchenko commented Dec 15, 2021 •
edited
Loading

What does this PR do?

  • provide a possibility to configure additional parsers for kubernetes container logs: ndjson and multiline
  • add sample_event and add missing fields
  • add possibility to configure container parser with format and stream configuration parameters

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Related issues

Screenshots

Screenshot 2021-12-16 at 13 57 25

Screenshot 2021-12-16 at 14 01 57

Screenshot 2021-12-16 at 14 21 20

Example of plain txt log:

Example of nginx plain text log 
{
  "_index": ".ds-logs-kubernetes.container_logs-default-2021.12.16-000001",
  "_id": "IWVnw30Bf7XRDSjVGObL",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "k8s.gcr.io/kube-proxy:v1.21.1"
      },
      "runtime": "containerd",
      "id": "f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f"
    },
    "kubernetes": {
      "container": {
        "name": "kube-proxy"
      },
      "node": {
        "uid": "57ccd748-c877-4be9-9b0e-568e9f205025",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "8ff288bc-5508-4232-9c69-5c1bad7bbaca",
        "ip": "172.20.0.2",
        "name": "kube-proxy-gxzks"
      },
      "namespace": "kube-system",
      "namespace_uid": "a4453575-518e-4a21-9909-34874f674177",
      "daemonset": {
        "name": "kube-proxy"
      },
      "namespace_labels": {
        "kubernetes_io/metadata_name": "kube-system"
      },
      "labels": {
        "controller-revision-hash": "6bc6858f58",
        "pod-template-generation": "1",
        "k8s-app": "kube-proxy"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "type": "filebeat",
      "ephemeral_id": "08146946-f5e2-44df-805b-709506fff10c",
      "version": "8.1.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/kube-proxy-gxzks_kube-system_kube-proxy-f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f.log"
      },
      "offset": 9863
    },
    "elastic_agent": {
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "version": "8.1.0",
      "snapshot": true
    },
    "message": "W1216 13:20:46.854871       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice",
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2021-12-16T13:20:46.855Z",
    "ecs": {
      "version": "8.0.0"
    },
    "stream": "stderr",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "kubernetes.container_logs"
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "5.10.47-linuxkit",
        "codename": "Core",
        "name": "CentOS Linux",
        "type": "linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.20.0.2",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "name": "kind-control-plane",
      "id": "85e35c2b5e1b39ba72393a6baf6ee7cd",
      "mac": [
        "4a:71:e6:e3:51:8a",
        "76:9b:c8:36:6f:9c",
        "42:c4:d2:ad:e0:3d",
        "02:42:ac:14:00:02",
        "02:42:ac:12:00:02"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2021-12-16T13:20:53Z",
      "dataset": "kubernetes.container_logs"
    }
  },
  "fields": {
    "kubernetes.node.uid": [
      "57ccd748-c877-4be9-9b0e-568e9f205025"
    ],
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.daemonset.name": [
      "kube-proxy"
    ],
    "elastic_agent.version": [
      "8.1.0"
    ],
    "kubernetes.namespace_uid": [
      "a4453575-518e-4a21-9909-34874f674177"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "4a:71:e6:e3:51:8a",
      "76:9b:c8:36:6f:9c",
      "42:c4:d2:ad:e0:3d",
      "02:42:ac:14:00:02",
      "02:42:ac:12:00:02"
    ],
    "container.id": [
      "f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "container.image.name": [
      "k8s.gcr.io/kube-proxy:v1.21.1"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "kube-system"
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "kubernetes.labels.k8s-app": [
      "kube-proxy"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.os.type": [
      "linux"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      9863
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.1.0"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.pod.uid": [
      "8ff288bc-5508-4232-9c69-5c1bad7bbaca"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.20.0.2",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2"
    ],
    "agent.type": [
      "filebeat"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "stream": [
      "stderr"
    ],
    "host.os.kernel": [
      "5.10.47-linuxkit"
    ],
    "kubernetes.pod.name": [
      "kube-proxy-gxzks"
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "host.id": [
      "85e35c2b5e1b39ba72393a6baf6ee7cd"
    ],
    "kubernetes.pod.ip": [
      "172.20.0.2"
    ],
    "kubernetes.container.name": [
      "kube-proxy"
    ],
    "elastic_agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "Core"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "kube-system"
    ],
    "kubernetes.labels.pod-template-generation": [
      "1"
    ],
    "message": [
      "W1216 13:20:46.854871       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice"
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2021-12-16T13:20:53.000Z"
    ],
    "@timestamp": [
      "2021-12-16T13:20:46.855Z"
    ],
    "host.os.platform": [
      "centos"
    ],
    "kubernetes.labels.controller-revision-hash": [
      "6bc6858f58"
    ],
    "log.file.path": [
      "/var/log/containers/kube-proxy-gxzks_kube-system_kube-proxy-f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f.log"
    ],
    "data_stream.dataset": [
      "kubernetes.container_logs"
    ],
    "agent.ephemeral_id": [
      "08146946-f5e2-44df-805b-709506fff10c"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "kubernetes.container_logs"
    ]
  }
}

Example of json logs

Example of elastic-agent json text log 
{
  "_index": ".ds-logs-kubernetes.container_logs-default-2021.12.16-000001",
  "_id": "1GVmw30Bf7XRDSjV_eVD",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "docker.elastic.co/beats/elastic-agent:8.1.0-SNAPSHOT"
      },
      "runtime": "containerd",
      "id": "47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21"
    },
    "kubernetes": {
      "container": {
        "name": "elastic-agent"
      },
      "node": {
        "uid": "57ccd748-c877-4be9-9b0e-568e9f205025",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "370d764e-f1b9-4932-934f-b415bae76b7d",
        "ip": "172.20.0.2",
        "name": "elastic-agent-tst5x"
      },
      "namespace": "kube-system",
      "daemonset": {
        "name": "elastic-agent"
      },
      "namespace_uid": "a4453575-518e-4a21-9909-34874f674177",
      "namespace_labels": {
        "kubernetes_io/metadata_name": "kube-system"
      },
      "labels": {
        "app": "elastic-agent",
        "controller-revision-hash": "77bd57578",
        "pod-template-generation": "3"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "ephemeral_id": "08146946-f5e2-44df-805b-709506fff10c",
      "type": "filebeat",
      "version": "8.1.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/elastic-agent-tst5x_kube-system_elastic-agent-47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21.log"
      },
      "offset": 105583
    },
    "elastic_agent": {
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "version": "8.1.0",
      "snapshot": true
    },
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2021-12-16T13:20:41.251Z",
    "ecs": {
      "version": "8.0.0"
    },
    "stream": "stderr",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "kubernetes.container_logs"
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "5.10.47-linuxkit",
        "codename": "Core",
        "name": "CentOS Linux",
        "type": "linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.20.0.2",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "name": "kind-control-plane",
      "id": "85e35c2b5e1b39ba72393a6baf6ee7cd",
      "mac": [
        "4a:71:e6:e3:51:8a",
        "76:9b:c8:36:6f:9c",
        "42:c4:d2:ad:e0:3d",
        "02:42:ac:14:00:02",
        "02:42:ac:12:00:02"
      ],
      "architecture": "x86_64"
    },
    "json": {
      "log.origin": {
        "file.line": 66,
        "file.name": "stateresolver/stateresolver.go"
      },
      "@timestamp": "2021-12-16T13:20:41.251Z",
      "ecs.version": "1.6.0",
      "log.level": "info",
      "message": "Updating internal state"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2021-12-16T13:20:46Z",
      "dataset": "kubernetes.container_logs"
    }
  },
  "fields": {
    "kubernetes.node.uid": [
      "57ccd748-c877-4be9-9b0e-568e9f205025"
    ],
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.daemonset.name": [
      "elastic-agent"
    ],
    "elastic_agent.version": [
      "8.1.0"
    ],
    "kubernetes.namespace_uid": [
      "a4453575-518e-4a21-9909-34874f674177"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "4a:71:e6:e3:51:8a",
      "76:9b:c8:36:6f:9c",
      "42:c4:d2:ad:e0:3d",
      "02:42:ac:14:00:02",
      "02:42:ac:12:00:02"
    ],
    "container.id": [
      "47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "container.image.name": [
      "docker.elastic.co/beats/elastic-agent:8.1.0-SNAPSHOT"
    ],
    "kubernetes.labels.app": [
      "elastic-agent"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "kube-system"
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.os.type": [
      "linux"
    ],
    "json.log.level": [
      "info"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      105583
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.1.0"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "json.@timestamp": [
      "2021-12-16T13:20:41.251Z"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.pod.uid": [
      "370d764e-f1b9-4932-934f-b415bae76b7d"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.20.0.2",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2"
    ],
    "agent.type": [
      "filebeat"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "stream": [
      "stderr"
    ],
    "host.os.kernel": [
      "5.10.47-linuxkit"
    ],
    "kubernetes.pod.name": [
      "elastic-agent-tst5x"
    ],
    "json.ecs.version": [
      "1.6.0"
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "json.log.origin.file.line": [
      66
    ],
    "host.id": [
      "85e35c2b5e1b39ba72393a6baf6ee7cd"
    ],
    "kubernetes.pod.ip": [
      "172.20.0.2"
    ],
    "kubernetes.container.name": [
      "elastic-agent"
    ],
    "elastic_agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "Core"
    ],
    "json.message": [
      "Updating internal state"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "kube-system"
    ],
    "kubernetes.labels.pod-template-generation": [
      "3"
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2021-12-16T13:20:46.000Z"
    ],
    "@timestamp": [
      "2021-12-16T13:20:41.251Z"
    ],
    "host.os.platform": [
      "centos"
    ],
    "kubernetes.labels.controller-revision-hash": [
      "77bd57578"
    ],
    "json.log.origin.file.name": [
      "stateresolver/stateresolver.go"
    ],
    "log.file.path": [
      "/var/log/containers/elastic-agent-tst5x_kube-system_elastic-agent-47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21.log"
    ],
    "data_stream.dataset": [
      "kubernetes.container_logs"
    ],
    "agent.ephemeral_id": [
      "08146946-f5e2-44df-805b-709506fff10c"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "kubernetes.container_logs"
    ]
  }
}

@elasticmachine
Copy link

elasticmachine commented Dec 15, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-12-17T11:31:24.736+0000

  • Duration: 30 min 27 sec

  • Commit: 37be361

Test stats 🧪

Test Results
Failed 0
Passed 116
Skipped 0
Total 116

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@tetianakravchenko
remove test; add container parser configuration; use generic Addition…
9f828c4 
…al parsers configuration instead of multilineParser ad jsonParser

Signed-off-by: Tetiana Kravchenko <[email protected]>
@tetianakravchenko
Copy link
Contributor Author

@mukeshelastic @akshay-saraswat please have a look on the screenshots in the PR description, and let me know if you have any objections on the UI for this feature

@tetianakravchenko tetianakravchenko added the enhancement New feature or request label Dec 16, 2021
@tetianakravchenko tetianakravchenko changed the title Support parsing json logs with kubernetes integration Support additional parser configuration: ndjson and multiline in container logs data-stream Dec 16, 2021
@tetianakravchenko
add a link to parsers documentation
783b598 
Signed-off-by: Tetiana Kravchenko <[email protected]>
@tetianakravchenko
Copy link
Contributor Author

we also discussed with @mukeshelastic option to enable json parser by default, for now was decided to keep it as a advanced configuration, as this configuration applies to all cluster and there is no conditions available to limit it to specified image/namespace. It might be changed later

@tetianakravchenko
Copy link
Contributor Author

/test

ChrsMark
ChrsMark reviewed Dec 17, 2021
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I left a few comments.

packages/kubernetes/changelog.yml Outdated
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.7.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an enhancement so I would go with 1.8.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done a156f50

packages/kubernetes/data_stream/container_logs/manifest.yml Outdated
default: |
# - ndjson:
# target: json
# - multiline: ~
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It this is the baseline config for multiline or just a placeholder? Maybe a more accurate default could be used here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider it more as a placeholder, to be honest I am not sure what could be a reasonable default here, any ideas here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe sth like

multiline.type: pattern
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

The very basic example from https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline. Of course this is not supposed to cover 100% the possible cases so it would be nice if we can refer to docs so as to be be more clear. The idea of having a more complete config here is to give a sense of how the config looks like and then users can dive more into the docs according their specific use cases.

Copy link
Contributor Author

@tetianakravchenko tetianakravchenko Dec 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChrsMark I think it makes sense to add it, at least to show how this will look like as a parser. I've added multiline config - 37be361
also run a quick test: after enabling

- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

it seems to work as expected (examle - kibana logs)
Screenshot 2021-12-17 at 12 30 26

packages/kubernetes/data_stream/container_logs/fields/base-fields.yml
@@ -51,6 +54,30 @@
description: >
Kubernetes hostname as reported by the node’s kernel

- name: node.labels.*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will also need to document node.annotations.* even if they are not generated by default and hence the tests would never warn us about having them undocumented. Same for namespace_annotations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done a156f50

ChrsMark
ChrsMark approved these changes Dec 17, 2021
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, good job here!

I left a small suggestion about the multiline, feel free to add it or reject it. Otherwise if CI is happy feel free to merge it!

@MichaelKatsoulis
Copy link
Contributor

Great job here @tetianakravchenko !

@tetianakravchenko
add multiline configuration example
37be361 
Signed-off-by: Tetiana Kravchenko <[email protected]>
@tetianakravchenko tetianakravchenko merged commit 1dbbe4e into elastic:master Dec 17, 2021
@tetianakravchenko tetianakravchenko mentioned this pull request Dec 23, 2021
Merged
6 tasks
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
@tetianakravchenko
Support additional parser configuration: ndjson and multiline in cont…
3e181ed 
…ainer logs data-stream (elastic#2345)

* add sample_event; support ndjson parser

Signed-off-by: Tetiana Kravchenko <[email protected]>

* add possibility to adjust container, ndjson and multiline parser configuratios

Signed-off-by: Tetiana Kravchenko <[email protected]>

* add pr link

Signed-off-by: Tetiana Kravchenko <[email protected]>

* remove test; add container parser configuration; use generic Additional parsers configuration instead of multilineParser ad jsonParser

Signed-off-by: Tetiana Kravchenko <[email protected]>

* add a link to parsers documentation

Signed-off-by: Tetiana Kravchenko <[email protected]>

* bump package version to 1.8.0; add node.annotations. and namespace_annotations.* fields

Signed-off-by: Tetiana Kravchenko <[email protected]>

* add container parser link to the documentation

Signed-off-by: Tetiana Kravchenko <[email protected]>

* add multiline configuration example

Signed-off-by: Tetiana Kravchenko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MichaelKatsoulis MichaelKatsoulis MichaelKatsoulis approved these changes

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support parsing json logs with kubernetes integration
4 participants
@tetianakravchenko @elasticmachine @MichaelKatsoulis @ChrsMark