Scrape etcd metrics without providing certificates

[govardha]

It is possible to scrape etcd metrics via the /metrics & /health endpoints when etcd is started up with the --listen-metrics set to 0.0.0.0.

I would think otel should be able to use this to scrape metrics rather than having to provide the certificates to otel.

https://etcd.io/docs/v3.4/op-guide/configuration/#--listen-metrics-urls

[jvoravong]

This helm chart's default configurations could be overridden to scrape metrics from the endpoints you mention if the endpoints were exposed properly.

Many Kubernetes distributions don't expose these endpoints by default. Some Kubernetes distributions may allow the user to expose these endpoints. Cloud based Kubernetes distributions don't typically allow you to collect metrics from the control plane where etcd resides. We don't track the status of Kubernetes distribution exposing etcd /metrics & /health endpoints. Check the documentation for your Kubernetes distribution about exposing these endpoints and using them.

[govardha]

I was able to figure this out, further I also had issues in scraping metrics in from other core k8s components. Right out of the box at least in my on prem cluster, I was only able to get coredns metrics. I wasn't getting any metrics from etcd/kubernetes-apiserver/kubernetes-proxy/kubernetes-controller-manager/kubernetes-scheduler.

The k8s community has changed how the metrics are exposed and the splunk-otel documentation is referring to the older way of scraping metrics. Further it looks like most people are using prometheus to start with, so this required a bit of digging around.

The kube-controller-manager wasn't listening in the standard http port(10252), but the https port(10257). so I came up with the block.

The intervalSeconds parameter is not documented anywhere which allows me to control how often the metrics are scraped.

Also keep in mind, the newer versions of etcd don't require certificates to scrape metrics, it has a command line argument called listen-metrics-url which allows you to scrape metrics without exposing the keys to the etcd kingdom.

Hope this helps out somebody else.

smartagent/kube-controller-manager:
  config:
     extraDimensions:
        metric_source: k8s-kcm
     port: 10257
     intervalSeconds: 30
     skipVerify: true
     type: kube-controller-manager
     useHTTPS: true
  rule: type == "pod" && labels["component"] == "kube-controller-manager"