Prometheus receiver does not scrape some Summary type metrics

[djjayeeta]

Describe the bug
A clear and concise description of what the bug is.

Steps to reproduce
If possible, provide a recipe for reproducing the error.

What did you expect to see?
I have a Prometheus receiver + file exporter + logging exporter config. I expect to see all gauge, histogram and summary metric in the json file from file exporter.

What did you see instead?
I can only see Gauge metrics, some of histogram and summary in the json file from fileexporter and logging exporter.

Specifically _sum, _count for summary metrics are not available

The Prometheus receiver translates counter, histogram, and summary metrics into OTLP metrics with non-cumulative temporality.

Is this the expected behavior with Promethues receiver? If so do I need modify/add a new fileexporter or the receiver to handle this scenario?

What version did you use?

otel/opentelemetry-collector-dev:latest@sha256:ceebae71600ece5b280f7b4f74e7dfde8670761cf84b5c2a5f17089e8db31be0

What config did you use?

receivers:
        prometheus:
          use_start_time_metric: false
          config:
            global:
              scrape_interval: 1m
            scrape_configs:
            - bearer_token_file:TOKEN
              job_name: kube-apiserver
              kubernetes_sd_configs:
              - role: pod
                selectors:
                - role: pod
                  field: "spec.nodeName=$NODE_NAME"
              params:
                component:
                - kube-apiserver
              metric_relabel_configs:
              - action: keep
                regex: (apiserver_admission_step_admission_duration_seconds_summary_sum)
                source_labels:
                - __name__
              relabel_configs:
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(.+)
              - action: replace
                regex: ([^:]+)(?::\d+)?
                replacement: "${ARG1}:9443"
                source_labels:
                - __address__
                target_label: __address__
              - action: keep
                regex: kube-apiserver
                source_labels:
                - __meta_kubernetes_pod_label_component
              - action: replace
                source_labels:
                - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_name
                target_label: pod
              scheme: https
              tls_config:
                cert_file: CERT_FILE
                insecure_skip_verify: true
                key_file: KEY_FILE
      exporters:
        file:
          path: /data/exporter/metrics-data.json
        logging:
          loglevel: debug
          sampling_initial: 5
          sampling_thereafter: 5
      service:
        pipelines:
          metrics/prom:
            receivers:
              - prometheus
            exporters:
              - logging
              - file

Environment
NA

Additional context
My metrics are exposed in prometheus format, so I am left with the choice of using prometheus receiver. The above config works for a prometheus server. So the metric is available.

cc @huyan0 @alolita @jmacd @bogdandrutu

[bogdandrutu]

This is a bug in the prometheus receiver, see #1255

[bogdandrutu]

I don't think the file exporter does not work, it exports the entire data as json, are you sure you receive the data?

[djjayeeta]

Yes, the logging exporter shows the histogram and summary metrics but I see them as DoubleDataPoints

opentelemetry-collector/exporter/loggingexporter/logging_exporter.go

Line 135 in 8422992

b.logEntry("DoubleDataPoints #%d", i)

[bogdandrutu]

That is not a histogram or summary it is a double data point.

[bogdandrutu]

Please add the logs.

[djjayeeta]

@bogdandrutu Sorry for the late response, it took me some time to debug. Both the logging exporter and file exporter is not able to export some summary type metrics, specially the _sum type

My config file

receivers:
        prometheus:
          use_start_time_metric: false
          config:
            global:
              scrape_interval: 1m
            scrape_configs:
            - bearer_token_file:TOKEN
              job_name: kube-apiserver
              kubernetes_sd_configs:
              - role: pod
                selectors:
                - role: pod
                  field: "spec.nodeName=$NODE_NAME"
              params:
                component:
                - kube-apiserver
              metric_relabel_configs:
              - action: keep
                regex: (apiserver_admission_step_admission_duration_seconds_summary_sum)
                source_labels:
                - __name__
              relabel_configs:
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(.+)
              - action: replace
                regex: ([^:]+)(?::\d+)?
                replacement: "${ARG1}:9443"
                source_labels:
                - __address__
                target_label: __address__
              - action: keep
                regex: kube-apiserver
                source_labels:
                - __meta_kubernetes_pod_label_component
              - action: replace
                source_labels:
                - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_name
                target_label: pod
              scheme: https
              tls_config:
                cert_file: CERT_FILE
                insecure_skip_verify: true
                key_file: KEY_FILE
      exporters:
        file:
          path: /data/exporter/metrics-data.json
        logging:
          loglevel: debug
          sampling_initial: 5
          sampling_thereafter: 5
      service:
        pipelines:
          metrics/prom:
            receivers:
              - prometheus
            exporters:
              - logging
              - file

With the same config I launched a prometheus server which is able to scrape the metric. I think this is issue of promethues receiver, which is not able to scrape apiserver_admission_step_admission_duration_seconds_summary_sum metric. I am not sure of the reason why

This is specially interesting as apiserver_admission_step_admission_duration_seconds_summary is scrapped by the receiver and exported properly. Am I missing something?

[djjayeeta]

I think I get it now, the _sum and _count metrics are summarized in the same metric https://github.com/open-telemetry/opentelemetry-collector/blob/master/receiver/prometheusreceiver/internal/metricfamily.go#L157.

Out of curiosity Is there a specific reason to do that rather than leaving the metric as it is. ex we get apiserver_admission_step_admission_duration_seconds_summary_sum as a separate metric.

[djjayeeta]

There is also a similar issue. I have a metric named X(gauge) and X_count(counter). Both metric have different set of labels. Due to the logic https://github.com/open-telemetry/opentelemetry-collector/blob/master/receiver/prometheusreceiver/internal/metricsbuilder.go#L199 they get merged in a single metric of gauge type(union of labels from both X and X_count), though the data points are not lost but it would require to custom logic to extract individual ones. Can we make this trimming optional? A boolean flag which will be default true and we can set it false for not trimming suffix.

I would also like to know the reasoning behind trimming suffixes. Is there some major drawback if we keep the suffix

[rmfitzpatrick]

I've run into this and think the issue is that summary types aren't supported by the internal pdata types: https://github.com/open-telemetry/opentelemetry-collector/blob/master/translator/internaldata/oc_to_metrics.go#L227. @bogdandrutu is there an inherent limitation to this or could support be added relatively easily? I'd be happy to take a crack at it if immediately feasible.

edit: this comment may apply primarily to #1255

[jmacd]

One proposal for Summary can be found here:
open-telemetry/opentelemetry-specification#982
(search for "A minor note about about Summary data points" in the description).

[JasonXZLiu]

@tigrannajaryan Would you be able to assign this to me? Thank you!
I'll add the summary metric to the logging exporter so we can close this issue.

[JasonXZLiu]

This issue is resolved (through PRs #2048 and #2084) and can be closed.