From 8c497e3962c989d1d74103294c1a5aaf6294bb69 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Mon, 17 Oct 2022 17:44:11 -0400 Subject: [PATCH 01/10] Support maps and heterogeneous arrays as attribute values Resolves https://github.com/open-telemetry/opentelemetry-specification/issues/376 Use cases where this is necessary or useful: 1. Specify more than one resource in the telemetry: https://github.com/open-telemetry/opentelemetry-specification/issues/579 2. Data coming from external source, e.g. AWS Metadata: https://github.com/open-telemetry/opentelemetry-specification/pull/596#issuecomment-628983680 or https://github.com/open-telemetry/opentelemetry-specification/issues/376#issuecomment-1227501082 3. Capturing HTTP headers: https://github.com/open-telemetry/opentelemetry-specification/issues/376#issuecomment-908493520 4. Structured stack traces: https://github.com/open-telemetry/opentelemetry-specification/pull/2841 5. Payloads as attributes: https://github.com/open-telemetry/oteps/pull/219#discussion_r971065645 This is a draft PR to see what the change looks like. If this PR is merged it will be nice to follow it up with: - A standard way of flattening maps and nested objects when converting from OTLP to formats that don't support maps/nested objects. - Recommendations for semantic conventions to use/not use complex objects. --- CHANGELOG.md | 3 +++ spec-compliance-matrix.md | 2 ++ specification/common/README.md | 17 +++++++++++++---- specification/trace/sdk_exporters/jaeger.md | 4 ++-- specification/trace/sdk_exporters/zipkin.md | 4 ++-- 5 files changed, 22 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index de8fc11a381..1769353155c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,9 @@ release. - Clarify that attribute keys are case-sensitive. ([#3784](https://github.com/open-telemetry/opentelemetry-specification/pull/3784)) +- Support maps and heterogeneous arrays as attribute values + ([#2888](https://github.com/open-telemetry/opentelemetry-specification/pull/2888)) + ### Supplementary Guidelines ## v1.28.0 (2023-12-07) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index abe1b80d33a..bb62153b4e0 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -215,6 +215,7 @@ formats is required. Implementing more than one format is optional. | A metric Producer accepts an optional metric Filter | | | | | | | | | | | | | | The metric Reader implementation supports registering metric Filter and passing them its registered metric Producers | | | | | | | | | | | | | | The metric SDK's metric Producer implementations uses the metric Filter | | | | | | | | | | | | | +| Non-homogeneous arrays and maps (including nested) | | | | | | | | | | | | | ## Logs @@ -242,6 +243,7 @@ Disclaimer: this list of features is still a work in progress, please refer to t | Feature | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift | |---------------------------------------------------------------------------------------------------------------------------------------------|----------|----|------|----|--------|------|--------|-----|------|-----|------|-------| | Create from Attributes | | + | + | + | + | + | + | + | + | + | + | + | +| Non-homogeneous arrays and maps (including nested) for attribute values | | | | | | | | | | | | | | Create empty | | + | + | + | + | + | + | + | + | + | + | + | | [Merge (v2)](specification/resource/sdk.md#merge) | | + | + | | + | + | + | + | + | + | + | | | Retrieve attributes | | + | + | + | + | + | + | + | + | + | + | + | diff --git a/specification/common/README.md b/specification/common/README.md index 4b9adc77b2d..390ddbb5d51 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -29,12 +29,21 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - The attribute key MUST be a non-`null` and non-empty string. - Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys. -- The attribute value is either: +- The attribute value can be of `any` type, where any is defined as one of the following: - A primitive type: string, boolean, double precision floating point (IEEE 754-1985) or signed 64 bit integer. - - An array of primitive type values. The array MUST be homogeneous, - i.e., it MUST NOT contain values of different types. + - A homogeneous array of values of primitive type [before 1.29.0]. + - An array of `any` values [since 1.29.0]. + - A key/value map, where key is string and value is `any` value [since 1.29.0]. -For protocols that do not natively support non-string values, non-string values SHOULD be represented as JSON-encoded strings. For example, the expression `int64(100)` will be encoded as `100`, `float64(1.5)` will be encoded as `1.5`, and an empty array of any type will be encoded as `[]`. +Complex attribute types (such as homogenous arrays, arrays of any, and maps) SHOULD be +used sparingly, in situations where their use minimizes manipulation of the data’s +original structure. + +When exporting to protocols that do not natively support a particular non-string +value type the value should be converted to a string JSON-encoding of the value. + +For example, the expression `int64(100)` will be encoded as `100`, `float64(1.5)` will +be encoded as `1.5`, and an empty array of any type will be encoded as `[]`. Attribute values expressing a numerical value of zero, an empty string, or an empty array are considered meaningful and MUST be stored and passed on to diff --git a/specification/trace/sdk_exporters/jaeger.md b/specification/trace/sdk_exporters/jaeger.md index 32d11f0ede1..6967e10d65a 100644 --- a/specification/trace/sdk_exporters/jaeger.md +++ b/specification/trace/sdk_exporters/jaeger.md @@ -138,8 +138,8 @@ OpenTelemetry Span `Attribute`(s) MUST be reported as `tags` to Jaeger. Primitive types MUST be represented by the corresponding types of Jaeger tags. -Array values MUST be serialized to string like a JSON list as mentioned in -[semantic conventions](../../overview.md#semantic-conventions). +Array and map values MUST be serialized to a JSON and recorded as a tag of string type +as mentioned in [attribute value definition](../../common/README.md#attribute). ### Links diff --git a/specification/trace/sdk_exporters/zipkin.md b/specification/trace/sdk_exporters/zipkin.md index 9a902619cc5..57a96e5f166 100644 --- a/specification/trace/sdk_exporters/zipkin.md +++ b/specification/trace/sdk_exporters/zipkin.md @@ -131,8 +131,8 @@ document maps to the strongly-typed fields of Zipkin spans. Primitive types MUST be converted to string using en-US culture settings. Boolean values MUST use lower case strings `"true"` and `"false"`. -Array values MUST be serialized to string like a JSON list as mentioned in -[semantic conventions](../../overview.md#semantic-conventions). +Array and map values MUST be serialized to a JSON and recorded as a tag of string type +as mentioned in [attribute value definition](../../common/README.md#attribute). TBD: add examples From c0ef00fc3bd935def201ee89e1270e3f97394c4b Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Wed, 10 Jan 2024 14:40:26 -0500 Subject: [PATCH 02/10] Changes to address PR comments --- specification/common/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specification/common/README.md b/specification/common/README.md index 390ddbb5d51..8c481ce3790 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -35,13 +35,14 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - An array of `any` values [since 1.29.0]. - A key/value map, where key is string and value is `any` value [since 1.29.0]. -Complex attribute types (such as homogenous arrays, arrays of any, and maps) SHOULD be -used sparingly, in situations where their use minimizes manipulation of the data’s -original structure. +Complex attribute types (such as arrays and maps): + +- SHOULD be used sparingly, in situations where their use minimizes manipulation of + the data’s original structure. +- SHOULD NOT be used with metrics. When exporting to protocols that do not natively support a particular non-string value type the value should be converted to a string JSON-encoding of the value. - For example, the expression `int64(100)` will be encoded as `100`, `float64(1.5)` will be encoded as `1.5`, and an empty array of any type will be encoded as `[]`. From c2d294118799ba7a59f016acc62f96e5d6a49ec0 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Wed, 10 Jan 2024 14:50:36 -0500 Subject: [PATCH 03/10] Add a note that reference loops are not allowed --- specification/common/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/common/README.md b/specification/common/README.md index 8c481ce3790..e16947c6d92 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -33,7 +33,8 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - A primitive type: string, boolean, double precision floating point (IEEE 754-1985) or signed 64 bit integer. - A homogeneous array of values of primitive type [before 1.29.0]. - An array of `any` values [since 1.29.0]. - - A key/value map, where key is string and value is `any` value [since 1.29.0]. + - A key/value map, where key is string and value is `any` value. Any form of reference + loops is disallowed. [since 1.29.0]. Complex attribute types (such as arrays and maps): From 1a385e7542780bee68603c13449151133d32a539 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 12:09:56 -0500 Subject: [PATCH 04/10] Completely prohibit complex types for metric attributes --- specification/common/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/specification/common/README.md b/specification/common/README.md index e16947c6d92..72086fb05c8 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -36,11 +36,15 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - A key/value map, where key is string and value is `any` value. Any form of reference loops is disallowed. [since 1.29.0]. -Complex attribute types (such as arrays and maps): +Complex attribute types (such as arrays and maps) SHOULD be used sparingly. They may +be useful for example in situations where their use minimizes manipulation +of the data’s original structure. -- SHOULD be used sparingly, in situations where their use minimizes manipulation of - the data’s original structure. -- SHOULD NOT be used with metrics. +The following value types MUST NOT be used as metric attribute values: + +- Non-homogeneous arrays (arrays containing values of different types). +- Nested arrays. +- Key/value maps. When exporting to protocols that do not natively support a particular non-string value type the value should be converted to a string JSON-encoding of the value. From b15695c2930c4299f7d7d43b2c1b1aa1be102428 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 12:14:04 -0500 Subject: [PATCH 05/10] Fixed a couple minor incorrect/missing bits --- specification/common/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/common/README.md b/specification/common/README.md index 72086fb05c8..81977d4da61 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -31,7 +31,7 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys. - The attribute value can be of `any` type, where any is defined as one of the following: - A primitive type: string, boolean, double precision floating point (IEEE 754-1985) or signed 64 bit integer. - - A homogeneous array of values of primitive type [before 1.29.0]. + - An array of values of primitive type [before 1.29.0]. - An array of `any` values [since 1.29.0]. - A key/value map, where key is string and value is `any` value. Any form of reference loops is disallowed. [since 1.29.0]. @@ -51,9 +51,9 @@ value type the value should be converted to a string JSON-encoding of the value. For example, the expression `int64(100)` will be encoded as `100`, `float64(1.5)` will be encoded as `1.5`, and an empty array of any type will be encoded as `[]`. -Attribute values expressing a numerical value of zero, an empty string, or an -empty array are considered meaningful and MUST be stored and passed on to -processors / exporters. +Attribute values expressing a numerical value of zero, an empty string, an +empty array or empty key/value map are considered meaningful and MUST be stored and +passed on to processors / exporters. Attribute values of `null` are not valid and attempting to set a `null` value is undefined behavior. From 5e0d08187e8cb234755656fe685bb4f988c71ef2 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 12:16:39 -0500 Subject: [PATCH 06/10] Use "cycle" instead of "loops" --- specification/common/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/common/README.md b/specification/common/README.md index 81977d4da61..b7ec9d566e7 100644 --- a/specification/common/README.md +++ b/specification/common/README.md @@ -34,7 +34,7 @@ An `Attribute` is a key-value pair, which MUST have the following properties: - An array of values of primitive type [before 1.29.0]. - An array of `any` values [since 1.29.0]. - A key/value map, where key is string and value is `any` value. Any form of reference - loops is disallowed. [since 1.29.0]. + cycle is disallowed. [since 1.29.0]. Complex attribute types (such as arrays and maps) SHOULD be used sparingly. They may be useful for example in situations where their use minimizes manipulation From 5ccaaf017a7d85fd0605f8e03f3b6d4dae50013e Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 12:18:27 -0500 Subject: [PATCH 07/10] Fix wording compliance matrix --- spec-compliance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index bb62153b4e0..c70ea152d11 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -215,7 +215,7 @@ formats is required. Implementing more than one format is optional. | A metric Producer accepts an optional metric Filter | | | | | | | | | | | | | | The metric Reader implementation supports registering metric Filter and passing them its registered metric Producers | | | | | | | | | | | | | | The metric SDK's metric Producer implementations uses the metric Filter | | | | | | | | | | | | | -| Non-homogeneous arrays and maps (including nested) | | | | | | | | | | | | | +| Non-homogeneous arrays and maps (including nested) for attribute values | | | | | | | | | | | | | ## Logs From dc31a4307aa13901fa2ad67ef63e7260062ff6d4 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 13:16:39 -0500 Subject: [PATCH 08/10] Add line for traces in compliance matrix --- spec-compliance-matrix.md | 1 + 1 file changed, 1 insertion(+) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index c70ea152d11..dd916cc5eda 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -65,6 +65,7 @@ formats is required. Implementing more than one format is optional. | Array of primitives (homogeneous) | | + | + | + | + | + | + | + | + | + | + | + | | `null` values documented as invalid/undefined | | + | + | + | + | + | N/A | + | | + | | N/A | | Unicode support for keys and string values | | + | + | + | + | + | + | + | + | + | + | + | +| Non-homogeneous arrays and maps (including nested) for attribute values | | | | | | | | | | | | | | [Span linking](specification/trace/api.md#specifying-links) | Optional | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift | | Links can be recorded on span creation | | + | + | | + | + | + | + | + | + | + | | | Links can be recorded after span creation | | | | | | | | | | + | | | From 4f5e141bbaa1078cd52b79fc7cf8f9e38ed98fb2 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Thu, 11 Jan 2024 13:18:17 -0500 Subject: [PATCH 09/10] Remove line item for metric in compliance matrix since it is not needed anymore (we prohibit complex types for metrics) --- spec-compliance-matrix.md | 1 - 1 file changed, 1 deletion(-) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index dd916cc5eda..d8565300073 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -216,7 +216,6 @@ formats is required. Implementing more than one format is optional. | A metric Producer accepts an optional metric Filter | | | | | | | | | | | | | | The metric Reader implementation supports registering metric Filter and passing them its registered metric Producers | | | | | | | | | | | | | | The metric SDK's metric Producer implementations uses the metric Filter | | | | | | | | | | | | | -| Non-homogeneous arrays and maps (including nested) for attribute values | | | | | | | | | | | | | ## Logs From 58cbb1a5a9c15aef41de21fb5541cb20232a5af9 Mon Sep 17 00:00:00 2001 From: Tigran Najaryan Date: Fri, 12 Jan 2024 10:32:30 -0500 Subject: [PATCH 10/10] Add line item for logs in compliance matrix --- spec-compliance-matrix.md | 1 + 1 file changed, 1 insertion(+) diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index d8565300073..3f0f80d8187 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -237,6 +237,7 @@ Disclaimer: this list of features is still a work in progress, please refer to t | OTLP File exporter | | | - | | - | | | | | | - | | | Can plug custom LogRecordExporter | | | + | | | | | + | | + | | | | Trace Context Injection | | | + | | + | | | + | | + | + | | +| Non-homogeneous arrays and maps (including nested) for attribute values| | | | | | | | | | | | | ## Resource