diff --git a/filebeat/docs/index.asciidoc b/filebeat/docs/index.asciidoc index dc306a37392..bb412da9734 100644 --- a/filebeat/docs/index.asciidoc +++ b/filebeat/docs/index.asciidoc @@ -20,6 +20,7 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :ignores_max_retries: :has_docker_label_ex: :has_modules_command: +:has_kubernetes_logs_path_matcher: :has_registry: :deb_os: :rpm_os: @@ -28,6 +29,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :docker_platform: :win_os: +:kubernetes_default_indexers: {docdir}/kubernetes-default-indexers-matchers.asciidoc + include::{libbeat-dir}/shared-beats-attributes.asciidoc[] include::./overview.asciidoc[] diff --git a/filebeat/docs/kubernetes-default-indexers-matchers.asciidoc b/filebeat/docs/kubernetes-default-indexers-matchers.asciidoc new file mode 100644 index 00000000000..3d7f8655cc6 --- /dev/null +++ b/filebeat/docs/kubernetes-default-indexers-matchers.asciidoc @@ -0,0 +1,14 @@ +When `add_kubernetes_metadata` is used with {beatname_uc}, it uses the +`container` indexer and the `logs_path`. So events whose path in `log.file.path` +contains a reference to a container ID are enriched with metadata of the pod of +this container. + +This behaviour can be disabled by disabling default indexers and matchers in the +configuration: +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_kubernetes_metadata: + default_indexers.enabled: false + default_matchers.enabled: false +------------------------------------------------------------------------------- diff --git a/libbeat/docs/processors-list.asciidoc b/libbeat/docs/processors-list.asciidoc index 4b8c4bf1b2c..bebffb9d30c 100644 --- a/libbeat/docs/processors-list.asciidoc +++ b/libbeat/docs/processors-list.asciidoc @@ -124,6 +124,7 @@ include::{libbeat-processors-dir}/add_id/docs/add_id.asciidoc[] endif::[] ifndef::no_add_kubernetes_metadata_processor[] include::{libbeat-processors-dir}/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc[] +include::{libbeat-processors-dir}/add_kubernetes_metadata/docs/indexers_and_matchers.asciidoc[] endif::[] ifndef::no_add_labels_processor[] include::{libbeat-processors-dir}/actions/docs/add_labels.asciidoc[] diff --git a/libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc b/libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc index bdc8bb05204..a46120a281c 100644 --- a/libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc +++ b/libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc @@ -24,19 +24,31 @@ The `add_kubernetes_metadata` processor has two basic building blocks which are: * Indexers * Matchers -Indexers take in a pod's metadata and builds indices based on the pod metadata. -For example, the `ip_port` indexer can take a Kubernetes pod and index the pod -metadata based on all `pod_ip:container_port` combinations. - -Matchers are used to construct lookup keys for querying indices. For example, -when the `fields` matcher takes `["metricset.host"]` as a lookup field, it would -construct a lookup key with the value of the field `metricset.host`. - +Indexers use pods metadata to create unique identifiers for each one of the +pods, these identifiers help to correlate the metadata of the observed pods with +actual events. For example, the `ip_port` indexer can take a Kubernetes pod and +create identifiers for it based on all its `pod_ip:container_port` combinations. + +Matchers use information in events to construct lookup keys that match the +identifiers created by the indexers. For example, when the `fields` matcher takes +`["metricset.host"]` as a lookup field, it would construct a lookup key with the +value of the field `metricset.host`. When one of this lookup keys match with one +of the identifiers, the event is enriched with the metadata of the identified +pod. + +ifdef::kubernetes_default_indexers[] +include::{kubernetes_default_indexers}[] +endif::kubernetes_default_indexers[] +ifndef::kubernetes_default_indexers[] Each Beat can define its own default indexers and matchers which are enabled by -default. For example, FileBeat enables the `container` indexer, which indexes +default. For example, Filebeat enables the `container` indexer, which identifies pod metadata based on all container IDs, and a `logs_path` matcher, which takes the `log.file.path` field, extracts the container ID, and uses it to retrieve metadata. +endif::kubernetes_default_indexers[] + +You can find more information about the available indexers and matchers, and some +examples in <>. The configuration below enables the processor when {beatname_lc} is run as a pod in Kubernetes. @@ -93,4 +105,4 @@ client. It defaults to `KUBECONFIG` environment variable if present. `default_indexers.enabled`:: (Optional) Enable/Disable default pod indexers, in case you want to specify your own. `default_matchers.enabled`:: (Optional) Enable/Disable default pod matchers, in -case you want to specify your own. \ No newline at end of file +case you want to specify your own. diff --git a/libbeat/processors/add_kubernetes_metadata/docs/indexers_and_matchers.asciidoc b/libbeat/processors/add_kubernetes_metadata/docs/indexers_and_matchers.asciidoc new file mode 100644 index 00000000000..4c7fdba7503 --- /dev/null +++ b/libbeat/processors/add_kubernetes_metadata/docs/indexers_and_matchers.asciidoc @@ -0,0 +1,112 @@ +[float] +[[kubernetes-indexers-and-matchers]] +=== Indexers and matchers + +==== Indexers + +Indexers use pods metadata to create unique identifiers for each one of the +pods. + +Available indexers are: + +`container`:: Identifies the pod metadata using the IDs of its containers. +`ip_port`:: Identifies the pod metadata using combinations of its IP and its exposed ports. +When using this indexer metadata is identified using the IP of the pods, and the +combination if `ip:port` for each one of the ports exposed by its containers. +`pod_name`:: Identifies the pod metadata using its namespace and its name as +`namespace/pod_name`. +`pod_uid`:: Identifies the pod metadata using the UID of the pod. + +==== Matchers + +Matchers are used to construct the lookup keys that match with the identifiers +created by indexes. + +===== `field_format` + +Looks up pod metadata using a key created with a string format that can include +event fields. + +This matcher has an option `format` to define the string format. This string +format can contain placeholders for any field in the event. + +For example, the following configuration uses the `ip_port` indexer to identify +the pod metadata by combinations of the pod IP and its exposed ports, and uses +the destination IP and port in events as match keys: + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_kubernetes_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - ip_port: + matchers: + - field_format: + format: '%{[destination.ip]}:%{[destination.port]}' +------------------------------------------------------------------------------- + +===== `fields` + +Looks up pod metadata using as key the value of some specific fields. When +multiple fields are defined, the first one included in the event is used. + +This matcher has an option `lookup_fields` to define the files whose value will +be used for lookup. + +For example, the following configuration uses the `ip_port` indexer to identify +pods, and defines a matcher that uses the destination IP or the server IP for the +lookup, the first it finds in the event: + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_kubernetes_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - ip_port: + matchers: + - fields: + lookup_fields: ['destination.ip', 'server.ip'] +------------------------------------------------------------------------------- + +ifdef::has_kubernetes_logs_path_matcher[] +===== `logs_path` + +Looks up pod metadata using identifiers extracted from the log path stored in +the `log.file.path` field. + +This matcher has the following configuration settings: + +`logs_path`:: (Optional) Base path of container logs. If not specified, it uses +the default logs path of the platform where {beatname_uc} is running. +`resource_type`:: (Optional) Type of the resource to obtain the ID of. It can be +`pod`, to make the lookup based on the pod UID, or `container`, to make the +lookup based on the container ID. It defaults to `container`. + +The default configuration is able to lookup the metadata using the container ID +when the logs are collected from the default docker logs path +(`/var/lib/docker/containers//...` on Linux). + +For example the following configuration would use the pod UID when the logs are +collected from `/var/lib/kubelet/pods//...`. + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_kubernetes_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - pod_uid: + matchers: + - logs_path: + logs_path: '/var/lib/kubelet/pods' + resource_type: 'pod' +------------------------------------------------------------------------------- +endif::has_kubernetes_logs_path_matcher[] diff --git a/metricbeat/docs/index.asciidoc b/metricbeat/docs/index.asciidoc index 088cc570c1f..95eea80c8b1 100644 --- a/metricbeat/docs/index.asciidoc +++ b/metricbeat/docs/index.asciidoc @@ -30,6 +30,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :no_decode_csv_fields_processor: :no_timestamp_processor: +:kubernetes_default_indexers: {docdir}/kubernetes-default-indexers-matchers.asciidoc + include::{libbeat-dir}/shared-beats-attributes.asciidoc[] include::./overview.asciidoc[] diff --git a/metricbeat/docs/kubernetes-default-indexers-matchers.asciidoc b/metricbeat/docs/kubernetes-default-indexers-matchers.asciidoc new file mode 100644 index 00000000000..2e1e4dde705 --- /dev/null +++ b/metricbeat/docs/kubernetes-default-indexers-matchers.asciidoc @@ -0,0 +1,14 @@ +When `add_kubernetes_metadata` is used with {beatname_uc}, it uses the `ip_port` +indexer and the `fields` matcher with the `metricset.host` field. So events that +contain a `metricset.host` field are enriched with metadata of the pods that +exposes the same port in the same IP. + +This behaviour can be disabled by disabling default indexers and matchers in the +configuration: +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_kubernetes_metadata: + default_indexers.enabled: false + default_matchers.enabled: false +-------------------------------------------------------------------------------