Clarify timestamps should not be added in prometheus export and remove current references

[anuraaga]

OpenMetrics spec gives explicit advice to not include timestamps in exported data.

https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exposing-timestamps

So we should probably follow this. Currently, we only mention timestamps for exemplars

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/datamodel.md#exemplars-2

We should remove this and add explicit text that timestamps MUST NOT be included to match the OpenMetrics guidance.

Note that timestamps can cause problems in the wild presumably due to Prometheus implementation limitations even when the format seems valid from an OpenMetrics standpoint.

open-telemetry/opentelemetry-java#4193

[anuraaga]

@open-telemetry/wg-prometheus Do you have any thoughts on this? Thanks

[dashpole]

There are use-cases in prometheus (e.g. collecting metrics from another metrics system), where timestamps should be used. IIRC, those aren't possible in with the current otel API (you can't specify that an observation occurred at some point in the past), so it makes sense to specify that SDK exporters MUST NOT expose timestamps. If otel gains the ability to record observations at a time in the past, we would want to expose an option to include timestamps. But for now, MUST NOT include timestamps in SDK exporters makes sense.

For the collector's prometheus exporter, it makes sense to include timestamps, since metrics may have been collected at a point further in the past. Including timestamps matches the behavior of the prometheus server's federation endpoint (I just checked).

[jsuereth]

I'm not sure I agree on exemplars unless prometheus doesn't support their timestamps correctly.

UNLIKE prometheus metrics where everything is cumulative, exemplars are snapshots of when something was recorded and the timestamp is as important as the value, particularly in cumulative instance where we may be carrying around stale exemplars.

[dashpole]

@anuraaga I think I understand your confusion now. The OpenMetrics guidance to not expose timestamps is only meant to apply to metric timestamps, not exemplar timestamps. We can, and should use timestamps with prometheus exemplars, but should not use timestamps for data points.

[dashpole]

From the OM spec:

in general, MetricPoint timestamps should not be exposed

The section is only talking about timestamps on MetricPoints, not on exemplars