Skip to content

Latest commit

 

History

History
106 lines (69 loc) · 6.97 KB

README.md

File metadata and controls

106 lines (69 loc) · 6.97 KB
Raw

Autodiscover provider in Elastic Agent

https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html

Kubernetes provider can be configured both in standalone and fleet managed elastic agent.

a. In standalone elastic agent, the kubernetes provider is configured inside the elastic-agent-standalone-kubernetes.yaml. User can update the parameters based on their needs.

b. In fleet managed elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the documentation to configure it.

Condition based autodiscover is supported both in fleet managed elastic agent and in standalone elastic agent (see doc).

Hints based autodiscover is only supported in standalone elastic agent (see doc).

Conditions based autodiscover

As an example we will use the redis module.

To automatically identify a Redis Pod and monitor it with the Redis integration, add the following input configuration inside the Elastic Agent Standalone manifest:

# Add extra input blocks here, based on conditions
# so as to automatically identify targeted Pods and start monitoring them
# using a predefined integration. For instance:
- name: redis
  type: redis/metrics
  use_output: default
  meta:
    package:
      name: redis
      version: 0.3.6
  data_stream:
    namespace: default
  streams:
    - data_stream:
        dataset: redis.info
        type: metrics
      metricsets:
        - info
      hosts:
        - '${kubernetes.pod.ip}:6379'
      idle_timeout: 20s
      maxconn: 10
      network: tcp
      period: 10s
      condition: ${kubernetes.pod.labels.app} == 'redis'

What makes this input block dynamic are the variables under hosts block and the condition.

${kubernetes.pod.ip} and ${kubernetes.pod.labels.app} == 'redis'

High level description

The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables (list of variables) for each k8s resource with unique value for each one of them.

Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blocks of the configurations.

This means that for every mapping that the condition matches (kubernetes.pod.labels.app == redis), a new input will be created in which the condition will be removed(not needed anymore) and the kubernetes.pod.ip variable will be substituted from the value in the same mapping.

The updated complete inputs block will be then forwarded to agent to spawn/update metricbeat and filebeat instances.

Hints based autodiscover

Standalone elastic agent supports autodiscover based on hints collected from the Kubernetes Provider. The hints mechanism looks for hints in kubernetes pod annotations that have the prefix co.elastic.hints. As soon as the Pod is ready, elastic agent checks it for hints and launches the proper configuration for the container. Hints tell elastic agent how to monitor the container by using the proper integration. The full list of supported hints can be found here.

Example:

As an example we will use again redis module. Add the following annotations to a redis pod. Elastic agent will then initiate a new input with redis module to properly monitor the redis pod.

apiVersion: v1 
kind: Pod
metadata:
  name: redis 
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info
    co.elastic.hints/info.period: 5m

In order to enable hints based autodiscover, user needs to uncomment the following lines in the Elastic Agent Standalone manifest

  1. Enable hints feature in kubernetes provider configuration.
  2. Add the init container that downloads the templates of various packages.
  3. Add the volumeMount and volume.

Note: If hints are enabled, the default_container_logs option is set to True by default. This means that logs from all discovered pods will be collected automatically, without requiring any annotations. To disable this behavior, set the option to False. This will stop the automatic log collection from all discovered pods. In this case, you must explicitly annotate your pods with the annotation co.elastic.hints/package: "container_logs" to collect their logs.

The init container will start before the elastic agent pod and will donwload all the templates of packages supported. Elastic agent will then collect from the pods running (through the watchers mechanism) all the hints annotations and will try to match them with the correct package. In the redis example, it will use the redis template and substitute the template variables with the values specified in the annotations. Default values will be used for variables not provided. A new input will be then created for redis and redis-specific data will start being collected.