From 5f391b25249f54aaf6b1f6237565ba3ac037ccce Mon Sep 17 00:00:00 2001 From: Georg Pirklbauer Date: Wed, 17 Aug 2022 16:43:36 +0200 Subject: [PATCH 1/4] Proposal for flattening attributes from OTLP messages --- specification/common/README.md | 4 + specification/common/attribute-precedence.md | 177 +++++++++++++++++++ 2 files changed, 181 insertions(+) create mode 100644 specification/common/attribute-precedence.md diff --git a/specification/common/README.md b/specification/common/README.md index 2b377a731cd..0e7e37e5df2 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -59,6 +59,10 @@ See [Requirement Level](attribute-requirement-level.md) for requirement levels g See [this document](attribute-type-mapping.md) to find out how to map values obtained outside OpenTelemetry into OpenTelemetry attribute values. +See [Attribute precedence for non-OTLP exporters](attribute-precedence.md) to +find out how to transform a structured representation like OTLP to a flat set of +unique attributes. + ### Attribute Limits Execution of erroneous code can result in unintended attributes. If there are no diff --git a/specification/common/attribute-precedence.md b/specification/common/attribute-precedence.md new file mode 100644 index 00000000000..2cb3f7b5cd7 --- /dev/null +++ b/specification/common/attribute-precedence.md @@ -0,0 +1,177 @@ +# Attribute precedence on transformation to non-OTLP formats + +**Status**: [Experimental](../document-status.md) + +
+Table of Contents + + + +- [Overview](#overview) +- [Attribute hierarchy in OTLP messages](#attribute-hierarchy-in-otlp-messages) +- [Precedence per Signal](#precedence-per-signal) + * [Traces](#traces) + * [Metrics](#metrics) + * [Logs](#logs) + * [Span Links, Span Events and Metric Exemplars](#span-links-span-events-and-metric-exemplars) +- [Considerations](#considerations) +- [Example](#example) +- [Useful links](#useful-links) + + + +
+ +## Overview + +This document provides supplementary guidelines for the attribute precedence +that exporters should follow when translating from the hierarchical OTLP format +to non-hierarchical formats. + +A mapping is required when flattening out attributes from the structured OTLP +format, which has attributes at different levels (e.g., Resource attributes, +InstrumentationScope attributes, attributes on Spans/Metrics/Logs) to a +non-hierarchical representation (e.g., Prometheus/OpenMetrics labels). +In the case of OpenMetrics, the set of labels is flat and must have unique +labels only +(). +Since OpenTelemetry allows for different levels of attributes, it is possible +that the same attribute appears multiple times on different levels. + +This document aims to provide guidance on how OpenTelemetry attributes can be +consistently mapped to flat sets. + +## Attribute hierarchy in OTLP messages + +Since the OTLP format is a hierarchical format, there is an inherent order in +the attributes. +In this document, +[Resource](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md) +attributes are considered to be at the top of the hierarchy, since they apply to +all collected telemetry. +Attributes on individual Spans/Metric data points/Logs are at the bottom of the +hierarchy, as they are most specialized and only apply to a subset of all data. + +**A more specialized attribute that shares an attribute key with more general +attribute will take precedence and overwrite the more general attribute.** + +When de-normalizing an OTLP message into a flat set of key-value pairs, +attributes that are present on the Resource and InstrumentationScope levels will +be attached to each Span/Metric data point/Log. + +## Precedence per Signal + +Below, the precedence for each of the signals is spelled out explicitly. +Only spans, metric data points and log records are considered. + +`A > B` denotes that the attribute on `A` will overwrite the attribute on `B` +if the keys clash. + +### Traces + +``` +Span.attributes > ScopeSpans.scope.attributes > ResourceSpans.resource.attributes +``` + +### Metrics + +Metrics are different from Spans and LogRecords, as each Metric has a data field +which can contain one or more data points. +Each data point has a set of attributes, which need to be considered +independently. + +``` +Metric.data.data_points.attributes > ScopeMetrics.scope.attributes > ResourceMetrics.resource.attributes +``` + +### Logs + +``` +LogRecord.log_records.attributes > ScopeLogs.scope.attributes > ResourceLogs.resource.attributes +``` + +### Span Links, Span Events and Metric Exemplars + +> Span Links, Span Events and Metric Exemplars need to be considered +> differently, as conflicting entries there can lead to problematic data loss. + +Consider a `http.host` attribute on a Span Link, which identifies the host of a +linked Span. +Following the "more specialized overwrites more general" suggestion leads to +overwriting the `http.host` attribute of the Span, which is likely desired +information. +Transferring attributes on Span Links, Span Events and Metric Exemplars should +be done separately from the parent Span/Metric data point. +This is out of the scope of these guidelines. + +## Considerations + +Note that this precedence is a strong suggestion, not a requirement. +Code that transforms attributes should follow this mode of flattening, but may +diverge if they have a reason to do so. + +## Example + +The following is a theoretical YAML-like representation of an OTLP message which +has attributes with attribute names that clash on multiple levels. + +```yaml +ResourceMetrics: + resource: + attributes: + # key-value pairs (attributes) on the resource + attribute1: resource-attribute-1 + attribute2: resource-attribute-2 + attribute3: resource-attribute-3 + service.name: my-service + + scope_metrics: + scope: + attributes: + attribute1: scope-attribute-1 + attribute2: scope-attribute-2 + attribute4: scope-attribute-4 + + metrics: + # there can be multiple data entries here. + data/0: + data_points: + # each data can have multiple data points: + data_point/1: + attributes: + # will overwrite scope and resource attribute + attribute1: data-point-1-attribute-1 + + data_point/2: + attributes: + # will overwrite + attribute1: data-point-2-attribute-1 + attribute4: data-point-2-attribute-4 +``` + +The structure above contains two data points, thus there will be two data points +in the output. +Their attributes will be: + +```yaml +# data point 1 +service.name: my-service # from the resource +attribute1: data-point-1-attribute-1 # overwrites attribute1 on resource & scope +attribute2: scope-attribute-2 # overwrites attribute2 on resource +attribute3: resource-attribute-3 # from the resource, not overwritten +attribute4: scope-attribute-4 # from the scope, not overwritten + +# data point 2 +service.name: my-service # from the resource +attribute1: data-point-2-attribute-1 # overwrites attribute1 on resource & scope +attribute2: scope-attribute-2 # overwrites attribute2 on resource +attribute3: resource-attribute-3 # from the resource, not overwritten +attribute4: data-point-2-attribute-4 # overwrites attribute4 from the scope +``` + +## Useful links + +* [Trace Proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto) +* [Metrics Proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) +* [Logs Proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto) +* [Resource Proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/resource/v1/resource.proto) From 66e04f3117af1ff7e75c4b3597ef36edfe4ea35a Mon Sep 17 00:00:00 2001 From: Georg Pirklbauer Date: Wed, 17 Aug 2022 16:59:43 +0200 Subject: [PATCH 2/4] add changelog entry --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e586aa94f0f..894e6d3fb91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,8 @@ release. - Introduce Instrumentation Scope Attributes ([#2579](https://github.com/open-telemetry/opentelemetry-specification/pull/2579)) +- Add supplementary guidance on attribute precedence when flattening out OTLP + ([#2736](https://github.com/open-telemetry/opentelemetry-specification/pull/2736)) ## v1.12.0 (2022-06-10) From c631230e1bd65a25923d8e00bdc9bf3d5624e77f Mon Sep 17 00:00:00 2001 From: Georg Pirklbauer Date: Mon, 5 Sep 2022 10:02:15 +0200 Subject: [PATCH 3/4] Update specification/common/attribute-precedence.md Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com> --- specification/common/attribute-precedence.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/common/attribute-precedence.md b/specification/common/attribute-precedence.md index 2cb3f7b5cd7..e141354a332 100644 --- a/specification/common/attribute-precedence.md +++ b/specification/common/attribute-precedence.md @@ -38,7 +38,7 @@ labels only Since OpenTelemetry allows for different levels of attributes, it is possible that the same attribute appears multiple times on different levels. -This document aims to provide guidance on how OpenTelemetry attributes can be +This document provides guidance on how OpenTelemetry attributes can be consistently mapped to flat sets. ## Attribute hierarchy in OTLP messages From 69d85013309afc6f379d10d352e925111edac083 Mon Sep 17 00:00:00 2001 From: Georg Pirklbauer Date: Fri, 23 Sep 2022 11:01:20 +0200 Subject: [PATCH 4/4] use zipkin as an example --- specification/common/attribute-precedence.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/specification/common/attribute-precedence.md b/specification/common/attribute-precedence.md index e141354a332..a4dcfb9963f 100644 --- a/specification/common/attribute-precedence.md +++ b/specification/common/attribute-precedence.md @@ -31,10 +31,8 @@ to non-hierarchical formats. A mapping is required when flattening out attributes from the structured OTLP format, which has attributes at different levels (e.g., Resource attributes, InstrumentationScope attributes, attributes on Spans/Metrics/Logs) to a -non-hierarchical representation (e.g., Prometheus/OpenMetrics labels). -In the case of OpenMetrics, the set of labels is flat and must have unique -labels only -(). +non-hierarchical representation (e.g., Zipkin tags). +In the case of Zipkin, tags are string-to-string dictionaries. Since OpenTelemetry allows for different levels of attributes, it is possible that the same attribute appears multiple times on different levels.