Status: Experimental
Table of Contents
This document covers OpenTelemetry compatibility with various Prometheus-related formats, including:
Formats used for Scraping metrics (pull):
- Prometheus text exposition format
- Prometheus protobuf format
- OpenMetrics text format
- (Not yet supported by Prometheus) OpenMetrics protobuf format
Formats used for Pushing metrics:
The document below uses "Prometheus" to refer to the union of all of these formats, even though support for particular features may be missing from the specific format. To avoid duplicating the specification for each format, this document will include requirements which may not be feasible to implement in all Prometheus formats. The following features are not consistently supported at the time of writing:
- Exemplars are not currently supported in the Prometheus text exposition format.
- Exemplars MUST be dropped if they are not supported.
- Info and StateSet-typed metrics are not currently supported by the Prometheus text exposition format or Prometheus protobuf format.
- If the specification below requires producing a Prometheus Info-typed metric, a Prometheus Gauge with an additional
_info
name suffix MUST be produced if Info-typed metrics are not supported. - If the specification below requires producing a Prometheus StateSet-typed metric, a Prometheus Gauge MUST be produced instead if StateSet-typed metrics are not supported.
- If the specification below requires producing a Prometheus Info-typed metric, a Prometheus Gauge with an additional
- Exponential (Native) Histograms are not currently supported in the Prometheus text exposition format or the OpenMetrics text or proto formats.
- Exponential (Native) Histograms SHOULD be dropped if they are not supported, or MAY be converted to fixed-bucket histograms.
The Prometheus Metric Name
MUST be added as the Name of the OTLP metric. By default, the name SHOULD NOT be altered, but translation SHOULD provide configuration which, when enabled, removes type (e.g. _total
) and unit (e.g. _seconds
) suffixes.
Prometheus UNIT metadata, if present, MUST be converted to the unit of the OTLP metric. The unit SHOULD be translated from Prometheus conventions to OpenTelemetry conventions by:
- Converting from full words to abbreviations (e.g. "milliseconds" to "ms").
- Special case: Converting "ratio" to "1".
- Converting "foo_per_bar" to "foo/bar".
Prometheus HELP metadata, if present, MUST be added as the description of the OTLP metric.
Prometheus TYPE metadata, if present, MUST be used to determine the OTLP data type, and dictates type-specific conversion rules listed below. Metric families without type metadata follow rules for unknown-typed metrics below.
A Prometheus Counter MUST be converted to an OTLP Sum with is_monotonic
equal to true
.
A Prometheus Gauge MUST be converted to an OTLP Gauge.
A Prometheus Info metric MUST be converted to an OTLP Non-Monotonic Sum unless it is the target
info metric, which is used to populate resource attributes. A Prometheus Info metric can be thought of as a special-case of the Prometheus Gauge metric which has a value of 1, and whose labels generally stays constant over the life of the process. It is converted to a OTLP Non-Monotonic Sum, rather than an OTLP Gauge, because the value of 1 is intended to be viewed as a count, which should be summed together when aggregating away labels.
A Prometheus StateSet metric MUST be converted to an OTLP Non-Monotonic Sum. A Prometheus StateSet metric can be thought of as a special-case of the Prometheus Gauge which has a 0 or 1 value, and has one metric point for every possible state. It is converted to an OTLP Non-Monotonic Sum, rather than an OTLP Gauge, because the value of 1 is intended to be viewed as a count, which should be summed together when aggregating away labels.
A Prometheus Unknown MUST be converted to an OTLP Gauge.
A Prometheus Histogram MUST be converted to an OTLP Histogram.
Multiple Prometheus histogram metrics MUST be merged together into a single OTLP Histogram:
- The
le
label on the_bucket
-suffixed metric is used to identify and order histogram bucket boundaries. Each Prometheus line produces one bucket count on the resulting histogram. Each value for thele
label except+Inf
produces one bucket boundary. - Lines with
_count
and_sum
suffixes are used to determine the histogram's count and sum. - If
_count
is not present, the metric MUST be dropped. - If
_sum
is not present, the histogram's sum MUST be unset.
Prometheus Summary MUST be converted to an OTLP Summary.
Multiple Prometheus metrics are merged together into a single OTLP Summary:
- The
quantile
label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus line produces one quantile on the resulting summary. - Lines with
_count
and_sum
suffixes are used to determine the summary's count and sum. - If
_count
is not present, the metric MUST be dropped. - If
_sum
is not present, the summary's sum MUST be set to zero.
The following Prometheus types MUST be dropped:
Prometheus Cumulative metrics can include the start time using the _created
sample series. When converting Prometheus Counters to OTLP, conversion SHOULD use _created
where available. When no _created
metric is available, conversion MUST follow Cumulative streams: handling unknown start time by default. Conversion MAY offer configuration, disabled by default, which allows using the process_start_time_seconds
metric to provide the start time. Using process_start_time_seconds
is only correct when all counters on the target start after the process and are not reset while the process is running.
Prometheus Exemplars
can be attached to Prometheus Histogram bucket metric points and counter metric
points. Exemplars on histogram buckets SHOULD be converted to exemplars on
OpenTelemetry histograms. Exemplars on counter metric points SHOULD be
converted to exemplars on OpenTelemetry sums. If present, the timestamp
MUST be added to the OpenTelemetry exemplar. The Trace ID and Span ID SHOULD be
retrieved from the trace_id
and span_id
label keys, respectively. All
labels not used for the trace and span ids MUST be added to the OpenTelemetry
exemplar as attributes.
The otel_scope_name
and otel_scope_version
lables, if present, SHOULD be
dropped from all metric points and used as the Instrumentation Scope name and
version respectively. All otel_scope_info
metrics present in a batch
of metrics SHOULD be dropped from the incoming scrape. Labels on
otel_scope_info
metric points other than otel_scope_name
and
otel_scope_version
, MUST be added as scope attributes to the scope with the
matching name and version. For example, the OpenMetrics text-formatted metrics:
# TYPE otel_scope_info info
otel_scope_info{otel_scope_name="go.opentelemetry.io.contrib.instrumentation.net.http.otelhttp",otel_scope_version="v0.24.0",library_mascot="bear"} 1
# TYPE http_server_duration counter
http_server_duration{otel_scope_name="go.opentelemetry.io.contrib.instrumentation.net.http.otelhttp",otel_scope_version="v0.24.0"...} 1
becomes:
# within a resource_metrics
scope_metrics:
scope:
name: go.opentelemetry.io.contrib.instrumentation.net.http.otelhttp
version: v0.24.0
attributes:
library_mascot: bear
metrics:
- name: http_server_duration
data:
sum:
data_points:
- value: 1
Metrics which do not have an otel_scope_name
or otel_scope_version
label
MUST be assigned an instrumentation scope identifying the entity performing
the translation from Prometheus to OpenTelemetry (e.g. the collector's
prometheus receiver).
When scraping a Prometheus endpoint, resource attributes MUST be added to the
scraped metrics to distinguish them from metrics from other Prometheus
endpoints. In particular, service.name
and service.instance.id
, are needed
to ensure Prometheus exporters can disambiguate metrics using
job
and instance
labels
as described below.
The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes:
OTLP Resource Attribute | Description |
---|---|
service.name |
The configured name of the service that the target belongs to |
service.instance.id |
A unique identifier of the target. By default, it should be the <host>:<port> of the scraped URL |
The following attributes SHOULD be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes:
OTLP Resource Attribute | Description |
---|---|
server.address |
The <host> portion of the target's URL that was scraped |
server.port |
The <port> portion of the target's URL that was scraped |
url.scheme |
http or https |
In addition to the attributes above, the
target
info metric is used to supply additional resource attributes. If present,
the target
info metric MUST be dropped from the batch of metrics, and all labels from
the target
info metric MUST be converted to resource attributes
attached to all other metrics which are part of the scrape. By default, label
keys and values MUST NOT be altered (such as replacing _
with .
characters
in keys).
Prometheus Pull exporters MUST NOT allow duplicate UNIT, HELP, or TYPE comments for the same metric name to be returned in a single scrape of the Prometheus endpoint. Exporters MUST drop entire metrics to prevent conflicting TYPE comments, but SHOULD NOT drop metric points as a result of conflicting UNIT or HELP comments. Instead, all but one of the conflicting UNIT and HELP comments (but not metric points) SHOULD be dropped. If dropping a comment or metric points, the exporter SHOULD warn the user through error logging.
The Name of an OTLP metric MUST be added as the
Prometheus Metric Name,
with unit and type suffixes added as described below. The metric name is
required to match the regex: [a-zA-Z_:]([a-zA-Z0-9_:])*
. Invalid characters
in the metric name MUST be replaced with the _
character. Multiple
consecutive _
characters MUST be replaced with a single _
character.
The Unit of an OTLP metric point SHOULD be converted to the equivalent unit in Prometheus when possible. This includes:
- Converting from abbreviations to full words (e.g. "ms" to "milliseconds").
- Dropping the portions of the Unit within brackets (e.g. {packet}). Brackets MUST NOT be included in the resulting unit. A "count of foo" is considered unitless in Prometheus.
- Special case: Converting "1" to "ratio".
- Converting "foo/bar" to "foo_per_bar".
The resulting unit SHOULD be added to the metric as UNIT metadata and as a suffix to the metric name unless the metric name already contains the unit, or the unit MUST be omitted. The unit suffix comes before any type-specific suffixes.
The description of an OTLP metrics point MUST be added as HELP metadata.
The data point type of an OTLP metric MUST be added as TYPE metadata. It also dictates type-specific conversion rules listed below.
Prometheus exporters SHOULD generate an Info-typed
metric named otel_scope_info
for each Instrumentation Scope with non-empty
scope attributes. If present, Instrumentation Scope name
and version
MUST
be added as otel_scope_name
and otel_scope_version
labels. Scope attributes
MUST also be added as labels following the rules described in the
Metric Attributes
section below.
Prometheus exporters MUST add the scope name as the otel_scope_name
label and
the scope version as the otel_scope_version
label on all metric points by
default, based on the scope the original data point was nested in.
Prometheus exporters SHOULD provide a configuration option to disable the
otel_scope_info
metric and otel_scope_
labels.
An OpenTelemetry Gauge MUST be converted to a Prometheus Gauge.
OpenTelemetry Sums follows this logic:
- If the aggregation temporality is cumulative and the sum is monotonic, it MUST be converted to a Prometheus Counter.
- If the aggregation temporality is cumulative and the sum is non-monotonic, it MUST be converted to a Prometheus Gauge.
- If the aggregation temporality is delta and the sum is monotonic, it SHOULD be converted to a cumulative temporality and become a Prometheus Counter. The following behaviors are expected:
- The new data point type must be the same as the accumulated data point type.
- The new data point's start time must match the time of the accumulated data point. If not, see detecting alignment issues.
- Otherwise, it MUST be dropped.
If the metric name for monotonic Sum metric points does not end in a suffix of _total
a suffix of _total
MUST be added by default, otherwise the name MUST remain unchanged. Exporters SHOULD provide a configuration option to disable the addition of _total
suffixes.
Monotonic Sum metric points with StartTimeUnixNano
should export the {name}_created
metric as well.
An OpenTelemetry Histogram with a cumulative aggregation temporality MUST be converted to a Prometheus metric family with the following metrics:
- A single
{name}_count
metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. {name}_sum
metric denoting the sum field of the histogram, reported only if the sum is positive and monotonic. The sum is positive and monotonic when all buckets are positive. All attributes of the histogram point are converted to Prometheus labels.- A series of
{name}_bucket
metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted asle
is added denoting the bucket boundary. The label's value is the stringified floating point value of bucket boundaries, ordered from lowest to highest. The value of each point is the sum of the count of all histogram buckets up the the boundary reported in thele
label. These points will include a single exemplar that falls withinle
label and no otherle
labelled point. The final bucket metric MUST have an+Inf
threshold. - Histograms with
StartTimeUnixNano
set should export the{name}_created
metric as well.
OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MUST be dropped.
An OpenTelemetry Exponential Histogram with a cumulative aggregation temporality MUST be converted to a Prometheus Native Histogram as follows:
Scale
is converted to the Native HistogramSchema
. Currently, valid values forschema
are -4 <= n <= 8. IfScale
is > 8 then Exponential Histogram data points SHOULD be downscaled to a scale accepted by Prometheus (in range [-4,8]). Any data point unable to be rescaled to an acceptable range MUST be dropped.Count
is converted to Native HistogramCount
if theNoRecordedValue
flag is set tofalse
, otherwise, Native HistogramCount
is set to the Stale NaN value.Sum
is converted to the Native HistogramSum
ifSum
is set and theNoRecordedValue
flag is set tofalse
, otherwise, Native HistogramSum
is set to the Stale NaN value.TimeUnixNano
is converted to the Native HistogramTimestamp
after converting nanoseconds to milliseconds.ZeroCount
is converted directly to the Native HistogramZeroCount
.ZeroThreshold
, if set, is converted to the Native HistogramZeroThreshold
. Otherwise, it is set to the default value1e-128
.- The dense bucket layout represented by
Positive
bucket counts andOffset
is converted to the Native Histogram sparse layout represented byPositiveSpans
andPositiveDeltas
. The same holds for theNegative
bucket counts andOffset
. Note that Prometheus Native Histograms buckets are indexed by upper boundary while Exponential Histograms are indexed by lower boundary, the result being that the Offset fields are different-by-one. Min
andMax
are not used.StartTimeUnixNano
is not used.
OpenTelemetry Exponential Histogram metrics with the delta aggregation temporality are dropped.
An OpenTelemetry Summary MUST be converted to a Prometheus metric family with the following metrics:
- A single
{name}_count
metric denoting the count field of the summary. All attributes of the summary point are converted to Prometheus labels. {name}_sum
metric denoting the sum field of the summary, reported only if the sum is positive and monotonic. All attributes of the summary point are converted to Prometheus labels.- A series of
{name}
metric points that contain all attributes of the summary point recorded as labels. Additionally, a label, denoted asquantile
is added denoting a reported quantile point, and having its value be the stringified floating point value of quantiles (between 0.0 and 1.0), starting from lowest to highest, and all being non-negative. The value of each point is the computed value of the quantile point. - Summaries with
StartTimeUnixNano
set should export the{name}_created
metric as well.
OpenTelemetry Metric Attributes MUST be converted to
Prometheus labels.
String Attribute values are converted directly to Metric Attributes, and
non-string Attribute values MUST be converted to string attributes following
the attribute specification. Prometheus
metric label keys are required to match the following regex:
[a-zA-Z_]([a-zA-Z0-9_])*
. Metrics from OpenTelemetry with unsupported
Attribute names MUST replace invalid characters with the _
character.
Multiple consecutive _
characters MUST be replaced with a single _
character. This may cause ambiguity in scenarios where multiple similar-named
attributes share invalid characters at the same location. In such unlikely
cases, if multiple key-value pairs are converted to have the same Prometheus
key, the values MUST be concatenated together, separated by ;
, and ordered by
the lexicographical order of the original keys.
Exemplars on OpenTelemetry Histograms and Monotonic Sums SHOULD
be converted to Prometheus exemplars. Exemplars on other OpenTelemetry data
points MUST be dropped. For Prometheus Remote Write exporters, multiple exemplars are
able to be added to each bucket, so all exemplars SHOULD be converted. For
Prometheus pull endpoints, only a single exemplar is able to be added to each
bucket, so the largest exemplar from each bucket MUST be used, if attaching
exemplars. If no exemplars exist on a bucket, the highest exemplar from a lower
bucket MUST be used, even though it is a duplicate of another bucket's exemplar.
Prometheus Exemplars MUST use the trace_id
and span_id
keys for the trace
and span IDs, respectively. Timestamps MUST be added as timestamps on the
Prometheus exemplar, and filtered_attributes
MUST be added as labels on the
Prometheus exemplar unless they would exceed the
limit on characters.
In Prometheus exporters, an OpenTelemetry Resource SHOULD be converted to
a target
info metric
if the resource is not empty.
The Resource attributes MAY be copied to labels of exported metric families
if required by the exporter configuration, or MUST be dropped. The target
info metric MUST be an info-typed
metric whose labels MUST include the resource attributes, and MUST NOT include
any other labels.
In the Collector's Prometheus exporters, it is
possible for metrics from multiple targets to be sent together, so targets must
be disambiguated from one another. However, the Prometheus exposition format
and remote-write
formats do not include a notion of resource, and expect metric labels to
distinguish scraped targets. By convention, job
and instance
labels distinguish targets and are expected to be present on metrics exposed on
a Prometheus pull exporter (a "federated"
Prometheus endpoint) or pushed via Prometheus remote-write. In OpenTelemetry
semantic conventions, the service.name
, service.namespace
, and
service.instance.id
triplet is
required to be unique,
which makes them good candidates to use to construct job
and instance
. In
the collector Prometheus exporters, the service.name
and service.namespace
attributes MUST be combined as <service.namespace>/<service.name>
, or
<service.name>
if namespace is empty, to form the job
metric label. The
service.instance.id
attribute, if present, MUST be converted to the
instance
label; otherwise, instance
should be added with an empty value.
Other resource attributes SHOULD be converted to a
target
info metric, or MUST be dropped. The target
metric is an info-typed metric
whose labels MUST include the resource attributes, and MUST NOT include any
other labels other than job
and instance
. There MUST be at most one
target
info metric exported for each unique combination of job
and instance
.
If info-typed metric families are not yet supported by the language Prometheus client library, a gauge-typed metric family named target_info
with a constant value of 1 MUST be used instead.
To convert OTLP resource attributes to Prometheus labels, string Attribute values are converted directly to labels, and non-string Attribute values MUST be converted to string attributes following the attribute specification.