From eda74857d0dbeb760eaa6017827bbb09e6c6640c Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 22 Apr 2021 02:02:48 -0700 Subject: [PATCH 1/9] Add comments about StartTimeUnixNano --- opentelemetry/proto/metrics/v1/metrics.proto | 202 +++++++++++-------- 1 file changed, 113 insertions(+), 89 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index df7b0d985..3d3a5a040 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -54,8 +54,8 @@ message InstrumentationLibraryMetrics { // // - Metric is composed of a metadata and data. // - Metadata part contains a name, description, unit. -// - Data is one of the possible types (Gauge, Sum, Histogram, etc.). -// - DataPoint contains timestamps, labels, and one of the possible value type +// - Data is one of the possible types (Sum, Gauge, Histogram, Sumary). +// - DataPoint contains timestamps, attributes, and one of the possible value type // fields. // // Metric @@ -95,14 +95,103 @@ message InstrumentationLibraryMetrics { // |+-----+ | // +---------------------------+ // +// Each distinct type of DataPoint represents the output of a specific +// aggregation function, the result of applying the DataPoint's +// associated function of to one or more measurements. +// // All DataPoint types have three common fields: -// - Labels zero or more key-value pairs associated with the data point. -// - StartTimeUnixNano MUST be set to the start of the interval when the data's -// type includes an AggregationTemporality. This field is not set otherwise. -// - TimeUnixNano MUST be set to: -// - the moment when an aggregation is reported (independent of the -// aggregation temporality). -// - the instantaneous time of the event. +// - Attributes includes key-value pairs associated with the data point +// - TimeUnixNano is required, set to the end time of the aggregation +// - StartTimeUnixNano is optional, but strongly encouraged for DataPoints +// having an AggregationTemporality field, as discussed below. +// +// Both TimeUnixNano and StartTimeUnixNano values are expressed as +// UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970. +// +// # TimeUnixNano +// +// This field is required, having consistent interpretation across +// DataPoint types. TimeUnixNano is the moment corresponding to when +// the data point's aggregate value was captured. +// +// Data points with/ the 0 value for TimeUnixNano SHOULD be rejected +// by consumers. +// +// # StartTimeUnixNano +// +// This field indicates to consumers the start time for points with +// cumulative and delta AggregationTemporality, and it should be +// included whenever possible to support correct rate calculation and +// overlap detection. Although it may be omitted when the start time +// is truly unknown, setting StartTimeUnixNano is strongly encouraged. +// +// There are two reasons that streams with AggregationTemporality are +// reset: (1) a process restarts, and (2) the SDK "forgets" the stream +// and then re-initializes it. StartTimeUnixNano is critical for +// calculating correct rates for Monotonic Sum, Histogram, and Summary +// data points in both of these cases +// +// StartTimeUnixNano may be used with Gauge instruments to indicate +// when the measurement was captured, as opposed to the moment it was +// observed. +// +// Another use for this information is to prevent overlap when +// multiple writers accidentally write to a single stream--applies +// to all data points. +// +// # Unknown StartTimeUnixNano +// +// This protocol adopts the Prometheus heuristic for reset detection, +// and with it prescribes a solution for filling in StartTimeUnixNanos +// with as much information as possible to assist consumers with +// calculating rates and detecting overlap. +// +// The Prometheus heuristic indicates that a reset must have occured +// in a monotonic series whenever the value descends from one value to +// the next, in a timestamp-ordered sequence. +// +// To facilitate reset and overlap detection, when single-timestamped +// sample values are translated into OTLP, such as from the Prometheus +// Remote-Write protocol, use the following. +// +// ## Filling in StartTimeUnixNano +// +// To import a sequence of non-overlapping measurements into into OTLP +// using a stateful processor: +// +// 1. Keep a map, using the metric name and attributes as the key, containing +// a struct of three parts: +// - resetTimestamp: the first observation or the reset timestamp +// - previousTimestamp: the timestamp of the most recent observation +// - previousValue: a monotonic value +// 2. When a new point arrives and it is the first observation: +// - in the map, set resetTimestamp to the input timestamp +// - in the map, set previousValue to the input value +// - in the map, set previousTimestamp to resetTimestamp +// - in the output, set StartTimeUnixNano to resetTimestamp +// 3. For each subsequent point that arrives in the sequence: +// - if the sample value is less than previousValue, +// then set resetTimestamp to previousTimestamp +// - set previousValue to the input value +// - set previousTimestamp to the input timestamp +// - in the output, set StartTimeUnixNano to resetTimestamp +// +// This technique ensures that all output points have a meaningful +// StartTimeUnixNano. The first point the stream written by this +// writer has TimeUnixNano equal to StartTimeUnixNano, a point with +// zero duration. Consumers of this data are expected to recognize +// these zero-width points as stream resets where the true start time +// is unknown. +// +// Using StartTimeUnixNano values filled in this way, consumers are +// able to calculate rates correctly and easily, without +// reconstructing a sequence of ordered points to apply the Prometheus +// heuristic. Receivers of cumulative and delta metric data with +// unknown start time are strongly encouraged to apply this logic as +// while translating into OTLP, to avoid producing data points with +// AggregationTemporality and unknown StartTimeUnixNano. +// +// The map can be cleared periodically, in effect resetting streams. message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1; @@ -319,24 +408,11 @@ message IntDataPoint { // The set of labels that uniquely identify this timeseries. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1; - // start_time_unix_nano is the last time when the aggregation value was reset - // to "zero". For some metric types this is ignored, see data types for more - // details. - // - // The aggregation value is over the time interval (start_time_unix_nano, - // time_unix_nano]. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. - // - // Value of 0 indicates that the timestamp is unspecified. In that case the - // timestamp may be decided by the backend. + // StartTimeUnixNano is optional but strongly encouraged, see the + // the detiled comments above Metric. fixed64 start_time_unix_nano = 2; - // time_unix_nano is the moment when this aggregation value was reported. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. + // TimeUnixNano is required, see the detailed comments above Metric. fixed64 time_unix_nano = 3; // value itself. @@ -364,24 +440,11 @@ message NumberDataPoint { // This field will be removed in ~3 months, on July 1, 2021. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1 [deprecated = true]; - // start_time_unix_nano is the last time when the aggregation value was reset - // to "zero". For some metric types this is ignored, see data types for more - // details. - // - // The aggregation value is over the time interval (start_time_unix_nano, - // time_unix_nano]. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. - // - // Value of 0 indicates that the timestamp is unspecified. In that case the - // timestamp may be decided by the backend. + // StartTimeUnixNano is optional but strongly encouraged, see the + // the detiled comments above Metric. fixed64 start_time_unix_nano = 2; - // time_unix_nano is the moment when this aggregation value was reported. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. + // TimeUnixNano is required, see the detailed comments above Metric. fixed64 time_unix_nano = 3; // The value itself. A point is considered invalid when one of the recognized @@ -414,24 +477,11 @@ message IntHistogramDataPoint { // The set of labels that uniquely identify this timeseries. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1; - // start_time_unix_nano is the last time when the aggregation value was reset - // to "zero". For some metric types this is ignored, see data types for more - // details. - // - // The aggregation value is over the time interval (start_time_unix_nano, - // time_unix_nano]. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. - // - // Value of 0 indicates that the timestamp is unspecified. In that case the - // timestamp may be decided by the backend. + // StartTimeUnixNano is optional but strongly encouraged, see the + // the detiled comments above Metric. fixed64 start_time_unix_nano = 2; - // time_unix_nano is the moment when this aggregation value was reported. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. + // TimeUnixNano is required, see the detailed comments above Metric. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. This @@ -499,24 +549,11 @@ message HistogramDataPoint { // This field will be removed in ~3 months, on July 1, 2021. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1 [deprecated = true]; - // start_time_unix_nano is the last time when the aggregation value was reset - // to "zero". For some metric types this is ignored, see data types for more - // details. - // - // The aggregation value is over the time interval (start_time_unix_nano, - // time_unix_nano]. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. - // - // Value of 0 indicates that the timestamp is unspecified. In that case the - // timestamp may be decided by the backend. + // StartTimeUnixNano is optional but strongly encouraged, see the + // the detiled comments above Metric. fixed64 start_time_unix_nano = 2; - // time_unix_nano is the moment when this aggregation value was reported. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. + // TimeUnixNano is required, see the detailed comments above Metric. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. This @@ -576,24 +613,11 @@ message SummaryDataPoint { // This field will be removed in ~3 months, on July 1, 2021. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1 [deprecated = true]; - // start_time_unix_nano is the last time when the aggregation value was reset - // to "zero". For some metric types this is ignored, see data types for more - // details. - // - // The aggregation value is over the time interval (start_time_unix_nano, - // time_unix_nano]. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. - // - // Value of 0 indicates that the timestamp is unspecified. In that case the - // timestamp may be decided by the backend. + // StartTimeUnixNano is optional but strongly encouraged, see the + // the detiled comments above Metric. fixed64 start_time_unix_nano = 2; - // time_unix_nano is the moment when this aggregation value was reported. - // - // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January - // 1970. + // TimeUnixNano is required, see the detailed comments above Metric. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. From 7e9029bc0b92f660ef67f25cbeb077ecce6c44da Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Fri, 23 Apr 2021 12:52:11 -0700 Subject: [PATCH 2/9] Apply suggestions from code review Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com> --- opentelemetry/proto/metrics/v1/metrics.proto | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 3d3a5a040..f429eaa49 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -54,7 +54,7 @@ message InstrumentationLibraryMetrics { // // - Metric is composed of a metadata and data. // - Metadata part contains a name, description, unit. -// - Data is one of the possible types (Sum, Gauge, Histogram, Sumary). +// - Data is one of the possible types (Sum, Gauge, Histogram, Summary). // - DataPoint contains timestamps, attributes, and one of the possible value type // fields. // @@ -126,10 +126,10 @@ message InstrumentationLibraryMetrics { // is truly unknown, setting StartTimeUnixNano is strongly encouraged. // // There are two reasons that streams with AggregationTemporality are -// reset: (1) a process restarts, and (2) the SDK "forgets" the stream +// reset: (1) a process restarts, and (2) the process "forgets" the stream // and then re-initializes it. StartTimeUnixNano is critical for // calculating correct rates for Monotonic Sum, Histogram, and Summary -// data points in both of these cases +// data points in both of these cases. // // StartTimeUnixNano may be used with Gauge instruments to indicate // when the measurement was captured, as opposed to the moment it was @@ -164,7 +164,7 @@ message InstrumentationLibraryMetrics { // - resetTimestamp: the first observation or the reset timestamp // - previousTimestamp: the timestamp of the most recent observation // - previousValue: a monotonic value -// 2. When a new point arrives and it is the first observation: +// 2. When a new point arrives and it is the first observation of that particular key: // - in the map, set resetTimestamp to the input timestamp // - in the map, set previousValue to the input value // - in the map, set previousTimestamp to resetTimestamp From 5b6abcbdabdccc3c2e4d39ebb0b5112d3e2b88eb Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 26 Apr 2021 00:29:39 -0700 Subject: [PATCH 3/9] From review feedback --- opentelemetry/proto/metrics/v1/metrics.proto | 66 ++++++++++++++++---- 1 file changed, 53 insertions(+), 13 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index f429eaa49..374922f78 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -156,25 +156,33 @@ message InstrumentationLibraryMetrics { // // ## Filling in StartTimeUnixNano // -// To import a sequence of non-overlapping measurements into into OTLP -// using a stateful processor: -// -// 1. Keep a map, using the metric name and attributes as the key, containing -// a struct of three parts: +// Note: This is meant to provide guidance in importing data into OTLP +// that does not have start time and there is no other form of reset +// detection. +// +// To import timestamped cumulative measurements into into OTLP using +// a stateful processor, the input must be processed in order. A +// series of points that formed the beginning of the sequence, +// including an unknown number of resets, are assumed to be lost or +// previously recorded when the processor begins observing points. +// Then: +// +// 1. Keep a map, using the metric name, resource, and attributes as +// the key, containing a struct of three parts: // - resetTimestamp: the first observation or the reset timestamp // - previousTimestamp: the timestamp of the most recent observation // - previousValue: a monotonic value // 2. When a new point arrives and it is the first observation of that particular key: -// - in the map, set resetTimestamp to the input timestamp -// - in the map, set previousValue to the input value -// - in the map, set previousTimestamp to resetTimestamp -// - in the output, set StartTimeUnixNano to resetTimestamp +// - in the map, set resetTimestamp to the arriving timestamp +// - in the map, set previousTimestamp to the arriving timestamp +// - in the map, set previousValue to the arriving value +// - in the output, set StartTimeUnixNano to the (new) resetTimestamp // 3. For each subsequent point that arrives in the sequence: // - if the sample value is less than previousValue, // then set resetTimestamp to previousTimestamp -// - set previousValue to the input value -// - set previousTimestamp to the input timestamp -// - in the output, set StartTimeUnixNano to resetTimestamp +// - set previousValue to the arriving value +// - set previousTimestamp to the arriving timestamp +// - in the output, set StartTimeUnixNano to (current) resetTimestamp // // This technique ensures that all output points have a meaningful // StartTimeUnixNano. The first point the stream written by this @@ -191,7 +199,9 @@ message InstrumentationLibraryMetrics { // while translating into OTLP, to avoid producing data points with // AggregationTemporality and unknown StartTimeUnixNano. // -// The map can be cleared periodically, in effect resetting streams. +// Enties can safely be removed from the map, after they appear no +// longer used. This has the effect of entering a gap stream where +// the processor reset its memory of the start time. message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1; @@ -410,9 +420,15 @@ message IntDataPoint { // StartTimeUnixNano is optional but strongly encouraged, see the // the detiled comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 time_unix_nano = 3; // value itself. @@ -442,9 +458,15 @@ message NumberDataPoint { // StartTimeUnixNano is optional but strongly encouraged, see the // the detiled comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 time_unix_nano = 3; // The value itself. A point is considered invalid when one of the recognized @@ -479,9 +501,15 @@ message IntHistogramDataPoint { // StartTimeUnixNano is optional but strongly encouraged, see the // the detiled comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. This @@ -551,9 +579,15 @@ message HistogramDataPoint { // StartTimeUnixNano is optional but strongly encouraged, see the // the detiled comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. This @@ -615,9 +649,15 @@ message SummaryDataPoint { // StartTimeUnixNano is optional but strongly encouraged, see the // the detiled comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. + // + // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January + // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. From 03b2d36fc414612c3e49bbad7b545163fac7b24b Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 26 Apr 2021 10:30:59 -0700 Subject: [PATCH 4/9] more text about unbroken sequence of observations --- opentelemetry/proto/metrics/v1/metrics.proto | 46 +++++++++++++++----- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 374922f78..98e59ee14 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -119,11 +119,18 @@ message InstrumentationLibraryMetrics { // // # StartTimeUnixNano // +// StartTimeUnixNano in general allows detecting when a sequence of +// observations is unbroken. StartTimeUnixNano is set to the +// beginning of an unbroken sequence of observations (all point kinds +// except with delta aggregation temporality) or the end of an +// unbroken sequence of observations (point kinds with delta +// temporality). +// // This field indicates to consumers the start time for points with // cumulative and delta AggregationTemporality, and it should be -// included whenever possible to support correct rate calculation and -// overlap detection. Although it may be omitted when the start time -// is truly unknown, setting StartTimeUnixNano is strongly encouraged. +// included whenever possible to support correct rate calculation. +// Although it may be omitted when the start time is truly unknown, +// setting StartTimeUnixNano is strongly encouraged. // // There are two reasons that streams with AggregationTemporality are // reset: (1) a process restarts, and (2) the process "forgets" the stream @@ -131,20 +138,39 @@ message InstrumentationLibraryMetrics { // calculating correct rates for Monotonic Sum, Histogram, and Summary // data points in both of these cases. // -// StartTimeUnixNano may be used with Gauge instruments to indicate -// when the measurement was captured, as opposed to the moment it was -// observed. +// Another use for this information which applies to all point kinds +// is to prevent overlap when multiple writers accidentally write to a +// single stream. StartTimeUnixNano MAY be used for overlap detection +// for all point kinds. +// +// ## StartTimeUnixNano: Sums and Histograms +// +// Since these two point kinds have AggregationTemporality, the use of +// StartTimeUnixNano is strongly recommended to ensure correct rate +// calculation and prevent overlap. +// +// ## StartTimeUnixNano: Gauges +// +// The use of StartTimeUnixNano is optional, recommended for overlap +// protection and to express explicitly absent points. When used, +// StartTimeUnixNano should be set as for cumulative aggregation +// temporality, to the starting timestamp in an unbroken sequence of +// observations. +// +// ## StartTimeUnixNano: Summaries // -// Another use for this information is to prevent overlap when -// multiple writers accidentally write to a single stream--applies -// to all data points. +// Summary points are treated the same as for cumulative aggregation +// temporality, by convention. StartTimeUnixNano should be set to the +// starting timestamp in an unbroken sequence of observations. // // # Unknown StartTimeUnixNano // // This protocol adopts the Prometheus heuristic for reset detection, // and with it prescribes a solution for filling in StartTimeUnixNanos // with as much information as possible to assist consumers with -// calculating rates and detecting overlap. +// calculating rates and detecting overlap. This process can be +// applied to Sum and Histogram points with cumulative aggregation +// temporality, Gauge, and Summary points. // // The Prometheus heuristic indicates that a reset must have occured // in a monotonic series whenever the value descends from one value to From 3dfeba06d5b9617accad3cb95bdadd6a0758a1ce Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 26 Apr 2021 22:23:42 -0700 Subject: [PATCH 5/9] Gauge overlap: delta or cumulative --- opentelemetry/proto/metrics/v1/metrics.proto | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 98e59ee14..4475215d9 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -153,9 +153,10 @@ message InstrumentationLibraryMetrics { // // The use of StartTimeUnixNano is optional, recommended for overlap // protection and to express explicitly absent points. When used, -// StartTimeUnixNano should be set as for cumulative aggregation -// temporality, to the starting timestamp in an unbroken sequence of -// observations. +// StartTimeUnixNano should be set as for either cumulative or delta +// aggregation temporality, meaning the start timestamp can be set to +// the previous point's `StartTimeUnixNano` or the previous point's +// `TimeUnixNano` to encode an unbroken sequence of Gauge observations. // // ## StartTimeUnixNano: Summaries // From fd4d37422b5e9d7fd0743af1fb509f55ab431d5e Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 3 May 2021 12:37:45 -0700 Subject: [PATCH 6/9] Update for gauges --- opentelemetry/proto/metrics/v1/metrics.proto | 31 ++++++++++---------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 4475215d9..6c7016269 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -151,12 +151,12 @@ message InstrumentationLibraryMetrics { // // ## StartTimeUnixNano: Gauges // -// The use of StartTimeUnixNano is optional, recommended for overlap -// protection and to express explicitly absent points. When used, -// StartTimeUnixNano should be set as for either cumulative or delta -// aggregation temporality, meaning the start timestamp can be set to -// the previous point's `StartTimeUnixNano` or the previous point's -// `TimeUnixNano` to encode an unbroken sequence of Gauge observations. +// The use of StartTimeUnixNano is optional for Gauge points. This is +// recommended in cases where there is a risk of mis-interpretion caused +// by overlapping points (by a violation of the single-writer principle). +// +// When used, StartTimeUnixNano should be set to the start time of the +// process or the timestamp when the Gauge was first set. // // ## StartTimeUnixNano: Summaries // @@ -185,14 +185,15 @@ message InstrumentationLibraryMetrics { // // Note: This is meant to provide guidance in importing data into OTLP // that does not have start time and there is no other form of reset -// detection. +// detection. This applies to points with cumulative aggregation +// temporality and Gauge points but cannot be applied to points with +// delta aggregation temporality. // -// To import timestamped cumulative measurements into into OTLP using -// a stateful processor, the input must be processed in order. A -// series of points that formed the beginning of the sequence, -// including an unknown number of resets, are assumed to be lost or -// previously recorded when the processor begins observing points. -// Then: +// To import timestamped measurements into into OTLP using a stateful +// processor, the input must be processed in order. A series of +// points that formed the beginning of the sequence, including an +// unknown number of resets, are assumed to be lost or previously +// recorded when the processor begins observing points. Then: // // 1. Keep a map, using the metric name, resource, and attributes as // the key, containing a struct of three parts: @@ -205,8 +206,8 @@ message InstrumentationLibraryMetrics { // - in the map, set previousValue to the arriving value // - in the output, set StartTimeUnixNano to the (new) resetTimestamp // 3. For each subsequent point that arrives in the sequence: -// - if the sample value is less than previousValue, -// then set resetTimestamp to previousTimestamp +// - for monotonic sum points, if the sample value is less than +// previousValue, then set resetTimestamp to previousTimestamp // - set previousValue to the arriving value // - set previousTimestamp to the arriving timestamp // - in the output, set StartTimeUnixNano to (current) resetTimestamp From bbe395927af2e8961a279fa290d22a77ab02048e Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 6 May 2021 16:59:37 -0700 Subject: [PATCH 7/9] Fixes --- opentelemetry/proto/metrics/v1/metrics.proto | 32 +++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 6c7016269..45f172b03 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -195,8 +195,9 @@ message InstrumentationLibraryMetrics { // unknown number of resets, are assumed to be lost or previously // recorded when the processor begins observing points. Then: // -// 1. Keep a map, using the metric name, resource, and attributes as -// the key, containing a struct of three parts: +// 1. Keep a map, using the metric name, resource, instrumentation +// library and attributes as the key, containing a struct of +// three parts: // - resetTimestamp: the first observation or the reset timestamp // - previousTimestamp: the timestamp of the most recent observation // - previousValue: a monotonic value @@ -204,7 +205,8 @@ message InstrumentationLibraryMetrics { // - in the map, set resetTimestamp to the arriving timestamp // - in the map, set previousTimestamp to the arriving timestamp // - in the map, set previousValue to the arriving value -// - in the output, set StartTimeUnixNano to the (new) resetTimestamp +// - in the output, set StartTimeUnixNano to the arriving timestamp +// (i.e., the current resetTimestamp). // 3. For each subsequent point that arrives in the sequence: // - for monotonic sum points, if the sample value is less than // previousValue, then set resetTimestamp to previousTimestamp @@ -277,7 +279,7 @@ message Metric { // IntGauge represents the type of a int scalar metric that always exports the // "current value" for every data point. It should be used for an "unknown" // aggregation. -// +// // A Gauge does not support different aggregation temporalities. Given the // aggregation is unknown, points cannot be combined using the same // aggregation, regardless of aggregation temporalities. Therefore, @@ -292,7 +294,7 @@ message IntGauge { // Gauge represents the type of a double scalar metric that always exports the // "current value" for every data point. It should be used for an "unknown" // aggregation. -// +// // A Gauge does not support different aggregation temporalities. Given the // aggregation is unknown, points cannot be combined using the same // aggregation, regardless of aggregation temporalities. Therefore, @@ -323,7 +325,7 @@ message IntSum { // as a sum of all reported measurements over a time interval. message Sum { repeated NumberDataPoint data_points = 1; - + // aggregation_temporality describes if the aggregator reports delta changes // since last report time, or cumulative changes since a fixed start time. AggregationTemporality aggregation_temporality = 2; @@ -431,7 +433,7 @@ enum AggregationTemporality { // number of requests received over the interval of time t_1 to // t_0+1 with a value of 1. // - // Note: Even though, when reporting changes since last report time, using + // Note: Even though, when reporting changes since last report time, using // CUMULATIVE is valid, it is not recommended. This may cause problems for // systems that do not use start_time to determine when the aggregation // value was reset (e.g. Prometheus). @@ -470,7 +472,7 @@ message IntDataPoint { // NumberDataPoint is a single data point in a timeseries that describes the // time-varying value of a double metric. message NumberDataPoint { - // The set of key/value pairs that uniquely identify the timeseries from + // The set of key/value pairs that uniquely identify the timeseries from // where this point belongs. The list may be empty (may contain 0 elements). repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; @@ -501,7 +503,7 @@ message NumberDataPoint { // value fields is not present inside this oneof. oneof value { double as_double = 4; - sfixed64 as_int = 6; + sfixed64 as_int = 6; } // (Optional) List of exemplars collected from @@ -517,7 +519,7 @@ message NumberDataPoint { // the distribution of those values across a set of buckets. // // If the histogram contains the distribution of values, then both -// "explicit_bounds" and "bucket counts" fields must be defined. +// "explicit_bounds" and "bucket counts" fields must be defined. // If the histogram does not contain the distribution of values, then both // "explicit_bounds" and "bucket_counts" must be omitted and only "count" and // "sum" are known. @@ -567,7 +569,7 @@ message IntHistogramDataPoint { // (-infinity, explicit_bounds[i]] for i == 0 // (explicit_bounds[i-1], explicit_bounds[i]] for 0 < i < N-1 // (explicit_bounds[i], +infinity) for i == N-1 - // + // // The values in the explicit_bounds array must be strictly increasing. // // Histogram buckets are inclusive of their upper boundary, except the last @@ -586,12 +588,12 @@ message IntHistogramDataPoint { // distribution of those values across a set of buckets. // // If the histogram contains the distribution of values, then both -// "explicit_bounds" and "bucket counts" fields must be defined. +// "explicit_bounds" and "bucket counts" fields must be defined. // If the histogram does not contain the distribution of values, then both // "explicit_bounds" and "bucket_counts" must be omitted and only "count" and // "sum" are known. message HistogramDataPoint { - // The set of key/value pairs that uniquely identify the timeseries from + // The set of key/value pairs that uniquely identify the timeseries from // where this point belongs. The list may be empty (may contain 0 elements). repeated opentelemetry.proto.common.v1.KeyValue attributes = 9; @@ -645,7 +647,7 @@ message HistogramDataPoint { // (-infinity, explicit_bounds[i]] for i == 0 // (explicit_bounds[i-1], explicit_bounds[i]] for 0 < i < N-1 // (explicit_bounds[i], +infinity) for i == N-1 - // + // // The values in the explicit_bounds array must be strictly increasing. // // Histogram buckets are inclusive of their upper boundary, except the last @@ -661,7 +663,7 @@ message HistogramDataPoint { // SummaryDataPoint is a single data point in a timeseries that describes the // time-varying values of a Summary metric. message SummaryDataPoint { - // The set of key/value pairs that uniquely identify the timeseries from + // The set of key/value pairs that uniquely identify the timeseries from // where this point belongs. The list may be empty (may contain 0 elements). repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; From ad79ff1bb527af5cd24077c59bb3ce1e677c4de6 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 10 May 2021 15:23:42 -0700 Subject: [PATCH 8/9] Remove detail text as being for the data model spec --- opentelemetry/proto/metrics/v1/metrics.proto | 112 +++---------------- 1 file changed, 16 insertions(+), 96 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 45f172b03..13726b98f 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -45,7 +45,11 @@ message InstrumentationLibraryMetrics { repeated Metric metrics = 2; } -// Defines a Metric which has one or more timeseries. +// Defines a Metric which has one or more timeseries. The following is a +// brief summary of the Metric data model. For more details, see: +// +// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/datamodel.md +// // // The data model and relation between entities is shown in the // diagram below. Here, "DataPoint" is the term used to refer to any @@ -114,46 +118,30 @@ message InstrumentationLibraryMetrics { // DataPoint types. TimeUnixNano is the moment corresponding to when // the data point's aggregate value was captured. // -// Data points with/ the 0 value for TimeUnixNano SHOULD be rejected +// Data points with the 0 value for TimeUnixNano SHOULD be rejected // by consumers. // // # StartTimeUnixNano // // StartTimeUnixNano in general allows detecting when a sequence of -// observations is unbroken. StartTimeUnixNano is set to the -// beginning of an unbroken sequence of observations (all point kinds -// except with delta aggregation temporality) or the end of an -// unbroken sequence of observations (point kinds with delta -// temporality). -// -// This field indicates to consumers the start time for points with -// cumulative and delta AggregationTemporality, and it should be -// included whenever possible to support correct rate calculation. -// Although it may be omitted when the start time is truly unknown, -// setting StartTimeUnixNano is strongly encouraged. -// -// There are two reasons that streams with AggregationTemporality are -// reset: (1) a process restarts, and (2) the process "forgets" the stream -// and then re-initializes it. StartTimeUnixNano is critical for -// calculating correct rates for Monotonic Sum, Histogram, and Summary -// data points in both of these cases. -// -// Another use for this information which applies to all point kinds -// is to prevent overlap when multiple writers accidentally write to a -// single stream. StartTimeUnixNano MAY be used for overlap detection -// for all point kinds. +// observations is unbroken. This field indicates to consumers the +// start time for points with cumulative and delta +// AggregationTemporality, and it should be included whenever possible +// to support correct rate calculation. Although it may be omitted +// when the start time is truly unknown, setting StartTimeUnixNano is +// strongly encouraged. // // ## StartTimeUnixNano: Sums and Histograms // // Since these two point kinds have AggregationTemporality, the use of // StartTimeUnixNano is strongly recommended to ensure correct rate -// calculation and prevent overlap. +// calculation and prevent the appearnce of overlapping points. // // ## StartTimeUnixNano: Gauges // // The use of StartTimeUnixNano is optional for Gauge points. This is // recommended in cases where there is a risk of mis-interpretion caused -// by overlapping points (by a violation of the single-writer principle). +// by overlapping points. // // When used, StartTimeUnixNano should be set to the start time of the // process or the timestamp when the Gauge was first set. @@ -161,77 +149,9 @@ message InstrumentationLibraryMetrics { // ## StartTimeUnixNano: Summaries // // Summary points are treated the same as for cumulative aggregation -// temporality, by convention. StartTimeUnixNano should be set to the +// temporality, by convention, allowing their `_sum` and `_count` fields +// to be used for rate calculations. StartTimeUnixNano should be set to the // starting timestamp in an unbroken sequence of observations. -// -// # Unknown StartTimeUnixNano -// -// This protocol adopts the Prometheus heuristic for reset detection, -// and with it prescribes a solution for filling in StartTimeUnixNanos -// with as much information as possible to assist consumers with -// calculating rates and detecting overlap. This process can be -// applied to Sum and Histogram points with cumulative aggregation -// temporality, Gauge, and Summary points. -// -// The Prometheus heuristic indicates that a reset must have occured -// in a monotonic series whenever the value descends from one value to -// the next, in a timestamp-ordered sequence. -// -// To facilitate reset and overlap detection, when single-timestamped -// sample values are translated into OTLP, such as from the Prometheus -// Remote-Write protocol, use the following. -// -// ## Filling in StartTimeUnixNano -// -// Note: This is meant to provide guidance in importing data into OTLP -// that does not have start time and there is no other form of reset -// detection. This applies to points with cumulative aggregation -// temporality and Gauge points but cannot be applied to points with -// delta aggregation temporality. -// -// To import timestamped measurements into into OTLP using a stateful -// processor, the input must be processed in order. A series of -// points that formed the beginning of the sequence, including an -// unknown number of resets, are assumed to be lost or previously -// recorded when the processor begins observing points. Then: -// -// 1. Keep a map, using the metric name, resource, instrumentation -// library and attributes as the key, containing a struct of -// three parts: -// - resetTimestamp: the first observation or the reset timestamp -// - previousTimestamp: the timestamp of the most recent observation -// - previousValue: a monotonic value -// 2. When a new point arrives and it is the first observation of that particular key: -// - in the map, set resetTimestamp to the arriving timestamp -// - in the map, set previousTimestamp to the arriving timestamp -// - in the map, set previousValue to the arriving value -// - in the output, set StartTimeUnixNano to the arriving timestamp -// (i.e., the current resetTimestamp). -// 3. For each subsequent point that arrives in the sequence: -// - for monotonic sum points, if the sample value is less than -// previousValue, then set resetTimestamp to previousTimestamp -// - set previousValue to the arriving value -// - set previousTimestamp to the arriving timestamp -// - in the output, set StartTimeUnixNano to (current) resetTimestamp -// -// This technique ensures that all output points have a meaningful -// StartTimeUnixNano. The first point the stream written by this -// writer has TimeUnixNano equal to StartTimeUnixNano, a point with -// zero duration. Consumers of this data are expected to recognize -// these zero-width points as stream resets where the true start time -// is unknown. -// -// Using StartTimeUnixNano values filled in this way, consumers are -// able to calculate rates correctly and easily, without -// reconstructing a sequence of ordered points to apply the Prometheus -// heuristic. Receivers of cumulative and delta metric data with -// unknown start time are strongly encouraged to apply this logic as -// while translating into OTLP, to avoid producing data points with -// AggregationTemporality and unknown StartTimeUnixNano. -// -// Enties can safely be removed from the map, after they appear no -// longer used. This has the effect of entering a gap stream where -// the processor reset its memory of the start time. message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1; From 55eb41700020df4389350463f09a9990282d8c0d Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 10 May 2021 16:19:58 -0700 Subject: [PATCH 9/9] Remove 134-154 --- opentelemetry/proto/metrics/v1/metrics.proto | 22 -------------------- 1 file changed, 22 deletions(-) diff --git a/opentelemetry/proto/metrics/v1/metrics.proto b/opentelemetry/proto/metrics/v1/metrics.proto index 13726b98f..09e6733b2 100644 --- a/opentelemetry/proto/metrics/v1/metrics.proto +++ b/opentelemetry/proto/metrics/v1/metrics.proto @@ -130,28 +130,6 @@ message InstrumentationLibraryMetrics { // to support correct rate calculation. Although it may be omitted // when the start time is truly unknown, setting StartTimeUnixNano is // strongly encouraged. -// -// ## StartTimeUnixNano: Sums and Histograms -// -// Since these two point kinds have AggregationTemporality, the use of -// StartTimeUnixNano is strongly recommended to ensure correct rate -// calculation and prevent the appearnce of overlapping points. -// -// ## StartTimeUnixNano: Gauges -// -// The use of StartTimeUnixNano is optional for Gauge points. This is -// recommended in cases where there is a risk of mis-interpretion caused -// by overlapping points. -// -// When used, StartTimeUnixNano should be set to the start time of the -// process or the timestamp when the Gauge was first set. -// -// ## StartTimeUnixNano: Summaries -// -// Summary points are treated the same as for cumulative aggregation -// temporality, by convention, allowing their `_sum` and `_count` fields -// to be used for rate calculations. StartTimeUnixNano should be set to the -// starting timestamp in an unbroken sequence of observations. message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1;