receiver/prometheus: Fix start timestamp adjustment logic #3047
Conversation
Signed-off-by: Anthony J Mirabella <[email protected]>
Signed-off-by: Anthony J Mirabella <[email protected]>
…tor into prom_start_time_fix
|
/cc @dashpole for GKE use-case |
|
I think this is the right thing to do based on the metrics data model specification from my reading of it, but i'm fairly sure we will need to make some adjustment so it isn't rejected by the googlecloud exporter. See census-ecosystem/opencensus-go-exporter-stackdriver#266 as an example. GCM doesn't seem to accept points with identical start and end times. |
|
Should those adjustments be made in the export path, or is there a requirement in OTLP that start time and end time are not equal? |
|
I don't think there is an OTLP requirement that they differ, but @bogdandrutu can correct me. When I say "we need to make an adjustment", I probably mean the stackdriver exporter needs to make an adjustment :) |
|
The opentelemetry go exporter for GCM already does some adjustment, but I don't see it in the opencensus exporter for GCM. |
|
I opened census-ecosystem/opencensus-go-exporter-stackdriver#287 to address this on stackdriver's end. I haven't tested it end-to-end, but I don't think it should block progress on this PR anymore. |
…tor into prom_start_time_fix
Signed-off-by: Anthony J Mirabella <[email protected]>
|
@Aneurysm9 can you resolve all the comments? |
…tor into prom_start_time_fix
Signed-off-by: Anthony J Mirabella <[email protected]>
|
@bogdandrutu I believe all comments have been resolved. |
…tor into prom_start_time_fix
…etry#3047) * receiver/prometheus: Do not drop metrics used to establish start time Signed-off-by: Anthony J Mirabella <[email protected]> * receiver/prometheus: update documentation regarding metric adjustment Signed-off-by: Anthony J Mirabella <[email protected]> * exporter/prometheus: fix e2e test to account for proper recevier functioning * Addres PR feedback on docs and naming Signed-off-by: Anthony J Mirabella <[email protected]> * rename adjustPoint->isReset Signed-off-by: Anthony J Mirabella <[email protected]>
…etry#3047) * receiver/prometheus: Do not drop metrics used to establish start time Signed-off-by: Anthony J Mirabella <[email protected]> * receiver/prometheus: update documentation regarding metric adjustment Signed-off-by: Anthony J Mirabella <[email protected]> * exporter/prometheus: fix e2e test to account for proper recevier functioning * Addres PR feedback on docs and naming Signed-off-by: Anthony J Mirabella <[email protected]> * rename adjustPoint->isReset Signed-off-by: Anthony J Mirabella <[email protected]>
…etry#3047) * receiver/prometheus: Do not drop metrics used to establish start time Signed-off-by: Anthony J Mirabella <[email protected]> * receiver/prometheus: update documentation regarding metric adjustment Signed-off-by: Anthony J Mirabella <[email protected]> * exporter/prometheus: fix e2e test to account for proper recevier functioning * Addres PR feedback on docs and naming Signed-off-by: Anthony J Mirabella <[email protected]> * rename adjustPoint->isReset Signed-off-by: Anthony J Mirabella <[email protected]>
Description: The Prometheus receiver has some logic that attempts to detect resets in metric series and augments scraped metrics with start timestamps based on its understanding of when a monotonic metric was last reset. The old logic involved storing the previous value for a timeseries and calculating a delta from that value, but it resulted in dropping the initial samples received for counters, histograms, and summaries as well as understating values from those metrics by an amount equal to the initial sample. The adjustment logic has been changed to only adjust the start timestamp if required and not change any metric values.
Testing: No tests were added, existing unit and e2e tests were updated to reflect the correct expectations.
Documentation: The
DESIGN.mddocument describing the receiver design was updated to reflect the modified start timestamp adjustment logic and for general copy editing.Fixes open-telemetry/prometheus-interoperability-spec#40.
Fixes open-telemetry/prometheus-interoperability-spec#42.
Fixes open-telemetry/prometheus-interoperability-spec#45.