[Prometheus Integration] Evaluate metrics extraction from Prometheus server using remote_write option with k8s

[gizas]

Currently Prometheus Integration is in 0.7.0 Technical Preview .

The goal of this issue is to evaluate current situation for the integration and evaluate the metric extraction from a Prometheus server using remote_write option with k8s.
In more details evaluate K8s metrics extraction by using such configuration.

Existing Prometheus Metricbeat Module:
https://www.elastic.co/guide/en/beats/metricbeat/8.2/metricbeat-module-prometheus.html

Evaluate:

• K8s metrics extraction
• OOTB Dashboard validation

Relevant tickets related with this:
a. #3487
b.#3489

[tetianakravchenko]

How remote_write works:
when prometheus.remove_write is enabled - metricbeat/elastic-agent start listening on the defined host:port (default: localhost:9201), Prometheus should use this address as a remote_write URL.

Issue in k8s: It is possible to run prometheus.remote_write in both modes: managed and standalone, the only missing part for both - we do not have a Service for elastic-agent to be able to refer it as a remove_write URL (prometheus config).

Current situation

We do already have prometheus integration https://github.com/elastic/integrations/tree/main/packages/prometheus, that supports collecting metrics from a Prometheus server using remote_write (remote_write data_stream).

Evaluation points

K8s metrics extraction:

How-to run managed elasti-agent:

1. Integration configuration:

2. Use next manifest - https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml and add Service resource:

---
apiVersion: v1
kind: Service
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  ports:
    - port: 9201
      protocol: TCP
      targetPort: 9201
  selector:
    app: elastic-agent
  sessionAffinity: None
  type: ClusterIP

2. Adjust FLEET_URL and FLEET_ENROLLMENT_TOKEN and apply manifests.
3. Install prometheus with modified configuration

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install prometheus-server prometheus-community/prometheus -f values.yml

$ cat values.yml
server:
  remoteWrite:
    - url: http://elastic-agent.kube-system:9201/write

Dashboard:

We have 1 Overview dashboard - it doesn't work

Few issue here/suggestion:

• Documentation is a bit confusing, as here is used localhost as an example. Used resources: https://www.elastic.co/guide/en/beats/metricbeat/8.2/metricbeat-metricset-prometheus-remote_write.html. Explanation on how it suppose to work would be useful to add (found it in https://www.elastic.co/blog/how-to-implement-prometheus-long-term-storage-using-elasticsearch)
• metrics are added under prometheus.* field:
  this contradicts with doc and do not align with collector metricset
• dashboard does not work, I think the point above is the main reason for that:
• After adding prometheus integration in UI, installation on K8s is not listed to enroll in Fleet
• Integration configuration in UI: some parameters could be hidden/moved to advanced settings
• in article mentioned dashboard for visualizing how many samples have been successfully sent to remote storage or how many have been retried, failed, or dropped. - adding smth like that in ootb could be useful

[tetianakravchenko]

Action points:

• Add documentation to test this scenario
• dashboard is not working: why metrics are written under prometheus.*? - created a follow-up issue [Prometheus integration] Refactor OOTB dashboard #3668
• dashboard with statics on number of scrapes/etc. - created a follow-up issue [Prometheus integration] Refactor OOTB dashboard #3668
• remote_write: Integration configuration in UI: some parameters could be hidden/moved to advanced settings - Fix gaps in the Prometheus integration (technical preview) #3662
• should Service resource be added ? - for now keep it only in documentation, no need to add to the manifest
• UI: Delete metricbeat how-to in integration - created follow-up issue - [Prometheus integration tutorial] Remove Metricbeat tutorial kibana#135981