From 5960d7f9cce1317cf15d08af1d5ef862149e8d38 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Tue, 30 Mar 2021 17:54:27 +0200 Subject: [PATCH] Add docs for Nomad autodiscover and add_nomad_metadata (#24557) Documentation for autodiscover was already there, but was not being included in the docs build. --- filebeat/docs/configuring-howto.asciidoc | 1 - filebeat/docs/index.asciidoc | 1 + libbeat/docs/processors-list.asciidoc | 6 + libbeat/docs/shared-autodiscover.asciidoc | 2 +- .../docs/add_nomad_metadata.asciidoc | 166 ++++++++++++++++++ 5 files changed, 174 insertions(+), 2 deletions(-) create mode 100644 x-pack/libbeat/processors/add_nomad_metadata/docs/add_nomad_metadata.asciidoc diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc index 4a2799c87aa..8bd08a838ef 100644 --- a/filebeat/docs/configuring-howto.asciidoc +++ b/filebeat/docs/configuring-howto.asciidoc @@ -60,7 +60,6 @@ include::./filebeat-filtering.asciidoc[] :autodiscoverHints: :autodiscoverNomad: include::{libbeat-dir}/shared-autodiscover.asciidoc[] -:autodiscoverNomad!: include::{libbeat-dir}/queueconfig.asciidoc[] diff --git a/filebeat/docs/index.asciidoc b/filebeat/docs/index.asciidoc index 69633f6836d..5193aef8667 100644 --- a/filebeat/docs/index.asciidoc +++ b/filebeat/docs/index.asciidoc @@ -22,6 +22,7 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :has_docker_label_ex: :has_modules_command: :has_kubernetes_logs_path_matcher: +:has_nomad_logs_path_matcher: :has_registry: :deb_os: :rpm_os: diff --git a/libbeat/docs/processors-list.asciidoc b/libbeat/docs/processors-list.asciidoc index e2670ebc39e..2d6965df500 100644 --- a/libbeat/docs/processors-list.asciidoc +++ b/libbeat/docs/processors-list.asciidoc @@ -29,6 +29,9 @@ endif::[] ifndef::no_add_locale_processor[] * <> endif::[] +ifndef::no_add_nomad_metadata_processor[] +* <> +endif::[] ifndef::no_add_observer_metadata_processor[] * <> endif::[] @@ -147,6 +150,9 @@ endif::[] ifndef::no_add_network_direction_processor[] include::{libbeat-processors-dir}/actions/docs/add_network_direction.asciidoc[] endif::[] +ifndef::no_add_nomad_metadata_processor[] +include::{x-libbeat-processors-dir}/add_nomad_metadata/docs/add_nomad_metadata.asciidoc[] +endif::[] ifndef::no_add_observer_metadata_processor[] include::{libbeat-processors-dir}/add_observer_metadata/docs/add_observer_metadata.asciidoc[] endif::[] diff --git a/libbeat/docs/shared-autodiscover.asciidoc b/libbeat/docs/shared-autodiscover.asciidoc index ba8fed0a2e6..7bfed6f8f74 100644 --- a/libbeat/docs/shared-autodiscover.asciidoc +++ b/libbeat/docs/shared-autodiscover.asciidoc @@ -505,7 +505,7 @@ ifdef::autodiscoverAWSEC2[] [float] ===== Amazon EC2s -*Note: This provider is experimental* +experimental[] The Amazon EC2 autodiscover provider discovers https://aws.amazon.com/ec2/[EC2 instances]. This is useful for users to launch {beatname_uc} modules to monitor services running on AWS EC2 instances. diff --git a/x-pack/libbeat/processors/add_nomad_metadata/docs/add_nomad_metadata.asciidoc b/x-pack/libbeat/processors/add_nomad_metadata/docs/add_nomad_metadata.asciidoc new file mode 100644 index 00000000000..1ab6b8cab03 --- /dev/null +++ b/x-pack/libbeat/processors/add_nomad_metadata/docs/add_nomad_metadata.asciidoc @@ -0,0 +1,166 @@ +[[add-nomad-metadata]] +[role="xpack"] +=== Add Nomad metadata + +++++ +add_nomad_metadata +++++ + +experimental[] + +The `add_nomad_metadata` processor adds fields with relevant metadata for +applications deployed in Nomad. + +Each event is annotated with the following information: + +* Allocation name, identifier and status. +* Job name and type. +* Namespace where the job is deployed. +* Datacenter and region where the agent runnning the allocation is located. + +[source,yaml] +------------------------------------------------------------------------------- +processors: + - add_nomad_metadata: ~ +------------------------------------------------------------------------------- + +It has the following settings to configure the connection: + +`address`:: (Optional) The URL of the agent API used to request the metadata. It +uses `http://127.0.0.1:4646` by default. +`namespace`:: (Optional) Namespace to watch. If set, only events for allocations +in this namespace will be annotated. +`region`:: (Optional) Region to watch. If set, only events for allocations in +this region will be annotated. +`secretID`:: (Optional) SecretID to use when connecting with the agent API. +`refresh_interval`:: (Optional) Interval used to updated the cached metadata. It +defaults to 30 seconds. +`cleanup_timeout`:: (Optional) After an allocation has been removed, time to +wait before cleaning up their associated resources. This is useful if you expect +to receive events after an allocation has been removed, what can happen when +collecting logs. It defaults to 60 seconds. + + +You can decide if {beatname_uc} should annotate events related to allocations in +local node or on the whole cluster configuring the scope with the following +settings: + +`scope`:: (Optional) Scope of the resources to watch. It can be `node` to get +metadata only for the allocations in a single agent, or `global`, to get metadata +for allocations running on any agent. It defaults to `node`. +`node`:: (Optional) When using `scope: node`, use `node` to specify the name of +the local node if it cannot be discovered automatically. + +For example the following configuration could be used if {beatname_uc} is +collecting events from all the allocations in the cluster: + +[source,yaml] +------------------------------------------------------------------------------- +processors: + - add_nomad_metadata: + scope: global +------------------------------------------------------------------------------- + +==== Indexers and matchers + +Indexers and matchers are used to correlate fields in events with actual +metadata. {beatname_uc} uses this information to know what metadata to include +in each event. + +===== Indexers + +Indexers use allocation metadata to create unique identifiers for each one of +the pods. + +Avaliable indexers are: +`allocation_name`:: Identifies allocations by its name and namespace (as +`/) +`allocation_uuid`:: Identifies allocations by its unique identifier. +// Review examples below when new indexers are added with network information. + +===== Matchers + +Matchers are used to construct the lookup keys that match with the identifiers +created by indexes. + +===== `field_format` + +Looks up allocation metadata using a key created with a string format that can include +event fields. + +This matcher has an option `format` to define the string format. This string +format can contain placeholders for any field in the event. + +For example, the following configuration uses the `allocation_name` indexer to identify +the allocation metadata by its name and namespace, and uses custom fields +existing in the event as match keys: + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_nomad_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - allocation_name: + matchers: + - field_format: + format: '%{[labels.nomad_namespace]}/%{[fields.nomad_alloc_name]}' +------------------------------------------------------------------------------- + +===== `fields` + +Looks up allocation metadata using as key the value of some specific fields. When +multiple fields are defined, the first one included in the event is used. + +This matcher has an option `lookup_fields` to define the fields whose value will +be used for lookup. + +For example, the following configuration uses the `allocation_uuid` indexer to +identify allocations, and defines a matcher that uses some fields where the +allocation UUID can be found for lookup, the first it finds in the event: + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_nomad_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - allocation_uuid: + matchers: + - fields: + lookup_fields: ['host.name', 'fields.nomad_alloc_uuid'] +------------------------------------------------------------------------------- + +ifdef::has_nomad_logs_path_matcher[] +===== `logs_path` + +Looks up allocation metadata using identifiers extracted from the log path stored in +the `log.file.path` field. + +This matcher has an optional `logs_path` option with the base path of the +directory containing the logs for the local agent. + +The default configuration is able to lookup the metadata using the allocation +UUID when the logs are collected under `/var/lib/nomad`. + +For example the following configuration would use the allocation UUID when the logs +are collected from `/var/lib/NomadClient001/alloc//alloc/logs/...`. + +[source,yaml] +------------------------------------------------------------------------------- +processors: +- add_nomad_metadata: + ... + default_indexers.enabled: false + default_matchers.enabled: false + indexers: + - allocation_uuid: + matchers: + - logs_path: + logs_path: '/var/lib/NomadClient001' +------------------------------------------------------------------------------- +endif::has_nomad_logs_path_matcher[]