[Elastic Agent] [Meta] Gathering logs and metrics from Kubernetes without using fleet generated agent config and without central agent management via fleet

[blakerouse]

Requirements

We expect Elastic Agent inside of Kubernetes in standalone mode (no connection to fleet-server, not using fleet-server to update agent config) to collect

• metrics and logs for kubernetes components - master node and the processes it runs, worker node and the processes it runs
• system metrics and logs of master and worker nodes
• pod metrics and logs to ES via auto-discovery through dynamic inputs and hints based discovery i.e. nginx running on K8s pods can send their metrics and logs to ES auto-magically as long as the agent policy contains appropriate conditional logic

In addition we will provide

• K8s integration that operator needs to install to observe K8s master and worker nodes and the infrastructure running K8s cluster, when installed it will display the K8s dashboards we provide today with metricbeat
• Documentation guide that walks through steps required to enable K8s and pod level observability with elastic agent in standalone mode

Note:
This issue explicitly ignores Fleet Mode, once standalone mode is working and documented then Fleet Mode will be the next target.

Core Issues

• Update K8s resource definition to use a host mount point for storing the running filebeat registry.
  • At the moment this is nested under the Elastic Agent data directory that changes on every build of Elastic Agent /usr/share/data/elastic-agent-{{ commit_short }}/run. Because of the {{commit_short}} being changed on every build this causes an issue with using a host mount point. [Elastic Agent] Allow separation of runtime state location and installation and executable location.  #24160
• Dynamic inputs fail to get validated [Agent] Dynamic inputs fail to get validated #23685
• Documentation guide that walks through steps required to enable K8s and pod level observability with elastic agent in standalone mode [Elastic Agent] Add documentation for the k8s provider #21848, PR: Add agent_on_k8s docs and k8s provider observability-docs#401
• Add Metricbeat --system.hostfs equivalent to Elastic Agent Add Metricbeat --system.hostfs equivalent to Elastic Agent #22915 PR: Add system.hostfs configuration option for system module #23831

Logs & Metrics

• System-level log collection - Logs from nodes of the Kubernetes cluster like /var/log/syslog', /var/log/secure`, etc. should be collected by Elastic Agent. PR: Add system level logs for Agent on k8s #24185
  • Update the K8s resource definition to include the correct host-level mount points to collect these logs.
  • Update the K8s resource definition to enable the system level log collection.
• Pod-level log collection - Logs from the running Pods in the cluster should be collected by Elastic Agent [Agent] Add agent standalone manifests for system module & Pod's log collection #23938
  • Update the K8s resource definition to include the correct host-level mount points to collect these logs.
  • Update the K8s resource definition to enable the log collection using inputs with variables.
• System-level metrics collection - Metrics (cpu/memory/etc.) from the nodes of the Kubernetes cluster [Agent] Add agent standalone manifests for system module & Pod's log collection #23938
  • Update the K8s resource definition to include the correct host-level mount points to collect these metrics from /proc and /sys.
    • Known Issue: Add Metricbeat --system.hostfs equivalent to Elastic Agent #22915
  • Update the K8s resource definition to enable the system metrics collection.
• Pod-level metrics collections - Metrics (cpu/memory/etc.) from the pods running in the Kubernetes cluster Add kubernetes manifests for standalone Agent with kubernetes integration  #23618
  • Update the K8s resource definition to collect metrics from kubelet.
• Kubernetes state metrics collection - Metrics (create/update/delete/etcs.) from the Kubernetes state Add kubernetes manifests for standalone Agent with kubernetes integration  #23618
  • Update the K8s resource definition to collect metrics from kube-state-metrics.
• K8s integrations package (template, dashboard) Migrate Kubernetes integration integrations#511

Need to be investigated

• hints support in K8s provider for agent Support hints based auto-discovery in kubernetes provider for elastic agent #23876

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[mukeshelastic]

@blakerouse thanks so much for writing this issue. I think this covers the K8s control plane observability for worker and master nodes very well. Does it also cover etcd observability? the pod level logs and metrics collection - what use cases does that cover? e.g. If pod runs nginx server then, are we able to collect nginx metrics and logs for it? What does the user need to do to enable that? Also if it's easier, I am happy to divide the issue into two separate ones - one for control plane observability and other for pod level observability.

[ChrsMark]

Hi @mukeshelastic ! If I'm not mistaken, Pod level observability will be covered by Dynamic Inputs and more specifically kubernetes provider (#21480), this is the former Autodiscover feature (hints not included yet). I tried to leverage this functionality at #23618 #23679 so as to dynamically collect metrics from k8s core components that are deployed as pods on master nodes. If we manage to make this use case work then we verify that we can support collecting metrics/logs from every pod dynamically, if it is covered by conditions in Agent's config since we don't support hints yet. I think we can add this as an extra/separate point in this Issue's description.

In this regard, we might need to cover this in our docs soon (#21848) cc: @ph

[ph]

@ChrsMark @blakerouse make sure if you find any issues related to the implementation please add them to this meta issue so we can prioritize them.

[ChrsMark]

@blakerouse about point no1 are we sure that we want it? We don't have something like this enabled by default right now in Filebeat's manifests.

cc: @masci @ruflin

[blakerouse]

@ChrsMark Are you talking about System-level log collection? I think it would be great to have. Log collection of the underlying host is just as important as collecting logs about the cluster and the workloads.

[ChrsMark]

@ChrsMark Are you talking about System-level log collection? I think it would be great to have. Log collection of the underlying host is just as important as collecting logs about the cluster and the workloads.

Yeap, ok then I agree it would be a good insight. Let's add it then.

[mukeshelastic]

There are few different layers for K8s here - system host that runs an OS, K8s worker node, pods, containers. And getting metrics and logs for each of them is valuable. Assuming your question was about "system host", yes it will be valuable to get those logs for troubleshooting system issues that may cause problems to pods running on it. I am assuming these logs are same as what we'd get by running system integration.

[ChrsMark]

@ruflin @ph @masci Related to ph's comment, since now we have most of the list completed what do you think about backporting the manifest to 7.13?

Sample PR: #24231

[ruflin]

++

[ph]

@ChrsMark +++ Please lets do it, this is super amazing. thanks for thepush on this.

[ChrsMark]

👍🏼 Back-ported to 7.x with #24231

[blakerouse]

There is only one issue left that is under "Needs Investigation":

• hints support in K8s provider for agent Support hints based auto-discovery in kubernetes provider for elastic agent #23876

I don't know if hints support is required for this to be considered to be complete, but we have made great progress on this issue.

[blakerouse]

Being that everything is fixed on this issue minus the hints supports, which has other requirements I am going to close this issue.