Improve Prometheus integration documentation

[tbragin]

Current docs on the Metricbeat Prometheus module are reference-style and quite minimal. The following blog expands on the way one can use this module and how it compares to other methods of integrating with Prometheus:
https://discuss.elastic.co/t/dec-23rd-2018-en-observability-querying-metrics-from-prometheus/161609

We should consider improving the docs, or linking to this blog.

cc: @roncohen @ruflin

[hodgesds]

Can you also include that exporting beat metrics to Prometheus is not supported.

[exekias]

Docs have improved after #9948: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-metricset-prometheus-collector.html.

@tbragin do you think this is enough or we should keep this issue?

[tbragin]

We are in the process of converting the discuss blog to a blog on elastic.co. How do folks feel about linking the reference-style Prometheus documentation to that blog? Also, do we feel we should evolve our modules documentation in general to include richer information, outside simply how to configure it? Perhaps I'll add a discuss label to this.

[tbragin]

cc: @makwarth @alvarolobato @DanRoscigno

[ruflin]

+1 on having links to prometheus docs. I think it's important to link to the reference docs.

++ on having richer module docs. All information a users needs to learn more about the module, how it works etc. should be there.

[makwarth]

++ absolutely. We probably need an owner per module that takes care of the Kibana and Elastic.co documentation

[DanRoscigno]

I have to test this, but I think the first YAML excerpt on the docs would be better like so (I was confused by the port 9090 on localhost because the example is supposed to be about grabbing from an exporter, not from the Prometheus server):

- module: prometheus
  period: 10s
  hosts:
    - nodehost:9100 <---- the existing port 9090 in the docs is confusing, as that is the Prometheus server port.  9100 is the default Node exporter port
    - haproxyhost:9101
    - mysqlhost:9104
  metrics_path: /metrics

[DanRoscigno]

@makwarth please bring me in when the discussion of what should be in module docs starts up. I approach this from the sysadmin point of view since I spent years in ops/services and in training people that sell to ops.

[DanRoscigno]

I opened this issue around scraping from exporters.

[DanRoscigno]

@tbragin @makwarth is the agenda set for EAH at Orlando? If there is time, can we have a slot to talk through what info Ops needs about a module? I have some strong opinions. Would be a good session for DeDe, Karen, dev, me, infra, SREs. If there is not time there, then we should do it after.

[tbragin]

@DanRoscigno You can suggest your topic here: https://docs.google.com/document/d/1-zyHzxk-s8MgqrjmV1OXI4TwP3vJmUkZSkVOHphNn4A/edit#

cc @alvarolobato Not sure if this would fall under plenary vs breakout.

[tbragin]

@sorantis I'd love your thoughts on this issue. We've had it open for a while, and some aspects have been addressed. Perhaps, as you you set up our Promethheus module yourself and walk through the experience, you could suggest where these reference docs can be shored up?

One improvement I can think of is documenting the expected document schema a bit better. I think it's one doc per poll interval per unique label combination (@exekias can confirm).

We should improve what we can and close soon.

[sorantis]

@tbragin sure, I can share my experience with the module so far.
One of the things I noticed when setting up Prometheus with two node_exporters scraped by the Metricbeat's Prometheus module, was that there was no straightforward way to filter metrics by node. When using multiple exporters Prometheus server adds additional labels to a metrics. E.g. for the two running node_exporters Prometheus added instance and job labels. Because these labels are added by the Prometheus server itself, they're not exposed by the service, and Metricbeat can't scrape them (thanks @exekias for explaining this). So, as a devops engineer, being used to Prometheus' labels, I find it hard to separate metrics per node_exporter in Metrics Explorer because the labels I'm relying upon are missing. Instead, as @exekias mentioned, the service.address filter should be used.
I think our documentation can mentioned this, or similar cases, where module's behavior diverges from expected one.
Another thing that IMHO is not clear is that based on Prometheus' data model (<metric name>{<label name>=<label value>, ...}) each metric has a set of labels associated with it. In case of our Prometheus module, labels and metrics are two separate fields, which should be intersected in order to replicate PromQL behavior. E.g. the results of the PromQL expression node_cpu_seconds_total{cpu="1", mode="idle"}, can be achieved in Metrics Explorer by selecting prometheus.metrics.node_cpu_seconds_total base metric and applying two additional filters on labels for cpu and mode.

Makes sense, but as someone who's just getting started with the module's capabilities, and has prior experience with Prometheus, I would've loved documentation for such implementation differences.

[roncohen]

@exekias could we add a labels.instance or similar in the Prometheus collector to make this a slightly smoother experience?

[exekias]

SGTM, I opened a new issue for that (#12739), let's move the discussion there and keep this one for docs.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.