From 070a8e4ff1b380ce1869d1eb9d7a53e9766ccb7d Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 2 Jul 2024 13:20:22 -0700 Subject: [PATCH 1/5] Rename prompt and completion tokens attributes to input and output --- docs/attributes-registry/gen-ai.md | 16 ++++++++++++++-- docs/gen-ai/gen-ai-spans.md | 4 ++-- model/registry/deprecated/gen-ai.yaml | 18 ++++++++++++++++++ model/registry/gen-ai.yaml | 4 ++-- model/trace/gen-ai.yaml | 4 ++-- schema-next.yaml | 5 +++++ 6 files changed, 43 insertions(+), 8 deletions(-) create mode 100644 model/registry/deprecated/gen-ai.yaml diff --git a/docs/attributes-registry/gen-ai.md b/docs/attributes-registry/gen-ai.md index 7030cb3bd3..e8b76d93cd 100644 --- a/docs/attributes-registry/gen-ai.md +++ b/docs/attributes-registry/gen-ai.md @@ -6,6 +6,9 @@ # Gen AI +- [Gen Ai](#gen-ai-attributes) +- [Gen Ai Deprecated](#gen-ai-deprecated-attributes) + ## Gen AI Attributes This document defines the attributes used to describe telemetry in the context of Generative Artificial Intelligence (GenAI) Models requests and responses. @@ -28,8 +31,8 @@ This document defines the attributes used to describe telemetry in the context o | `gen_ai.response.model` | string | The name of the model that generated the response. | `gpt-4-0613` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `gen_ai.system` | string | The Generative AI product as identified by the client or server instrumentation. [3] | `openai` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `gen_ai.token.type` | string | The type of token being counted. | `input`; `output` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `gen_ai.usage.completion_tokens` | int | The number of tokens used in the GenAI response (completion). | `180` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `gen_ai.usage.prompt_tokens` | int | The number of tokens used in the GenAI input or prompt. | `100` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `gen_ai.usage.input_tokens` | int | The number of tokens used in the GenAI input or prompt. | `100` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `gen_ai.usage.output_tokens` | int | The number of tokens used in the GenAI response (completion). | `180` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | **[1]:** It's RECOMMENDED to format completions as JSON string matching [OpenAI messages format](https://platform.openai.com/docs/guides/text-generation) @@ -53,3 +56,12 @@ For custom model, a custom friendly name SHOULD be used. If none of these option | -------- | ------------------------------------------ | ---------------------------------------------------------------- | | `input` | Input tokens (prompt, input, etc.) | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `output` | Output tokens (completion, response, etc.) | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +## Gen AI Deprecated Attributes + +Describes deprecated `gen_ai` attributes. + +| Attribute | Type | Description | Examples | Stability | +| -------------------------------- | ------ | ----------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `gen_ai.usage.completion_tokens` | string | Deprecated, use `gen_ai.usage.output_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.output_tokens` attribute. | +| `gen_ai.usage.prompt_tokens` | string | Deprecated, use `gen_ai.usage.input_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.input_tokens` attribute. | diff --git a/docs/gen-ai/gen-ai-spans.md b/docs/gen-ai/gen-ai-spans.md index d2fb54b987..a92b2c2a54 100644 --- a/docs/gen-ai/gen-ai-spans.md +++ b/docs/gen-ai/gen-ai-spans.md @@ -57,8 +57,8 @@ These attributes track input data and metadata for a request to an GenAI model. | [`gen_ai.response.finish_reasons`](/docs/attributes-registry/gen-ai.md) | string[] | Array of reasons the model stopped generating tokens, corresponding to each generation received. | `["stop"]` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | [`gen_ai.response.id`](/docs/attributes-registry/gen-ai.md) | string | The unique identifier for the completion. | `chatcmpl-123` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the model that generated the response. [3] | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| [`gen_ai.usage.completion_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI response (completion). | `180` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| [`gen_ai.usage.prompt_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI input or prompt. | `100` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| [`gen_ai.usage.input_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI input or prompt. | `100` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| [`gen_ai.usage.output_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI response (completion). | `180` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | **[1]:** The name of the GenAI model a request is being made to. If the model is supplied by a vendor, then the value must be the exact name of the model requested. If the model is a fine-tuned custom model, the value should have a more specific name than the base model that's been fine-tuned. diff --git a/model/registry/deprecated/gen-ai.yaml b/model/registry/deprecated/gen-ai.yaml new file mode 100644 index 0000000000..5d37bd3da8 --- /dev/null +++ b/model/registry/deprecated/gen-ai.yaml @@ -0,0 +1,18 @@ +groups: + - id: registry.gen_ai.deprecated + type: attribute_group + brief: Describes deprecated `gen_ai` attributes. + attributes: + - id: gen_ai.usage.prompt_tokens + type: string + stability: experimental + deprecated: Replaced by `gen_ai.usage.input_tokens` attribute. + brief: "Deprecated, use `gen_ai.usage.input_tokens` instead." + examples: [42] + - id: gen_ai.usage.completion_tokens + type: string + stability: experimental + deprecated: Replaced by `gen_ai.usage.output_tokens` attribute. + brief: "Deprecated, use `gen_ai.usage.output_tokens` instead." + examples: [42] + diff --git a/model/registry/gen-ai.yaml b/model/registry/gen-ai.yaml index 2bbf16e7c4..db474db703 100644 --- a/model/registry/gen-ai.yaml +++ b/model/registry/gen-ai.yaml @@ -90,12 +90,12 @@ groups: type: string[] brief: Array of reasons the model stopped generating tokens, corresponding to each generation received. examples: ['stop'] - - id: usage.prompt_tokens + - id: usage.input_tokens stability: experimental type: int brief: The number of tokens used in the GenAI input or prompt. examples: [100] - - id: usage.completion_tokens + - id: usage.output_tokens stability: experimental type: int brief: The number of tokens used in the GenAI response (completion). diff --git a/model/trace/gen-ai.yaml b/model/trace/gen-ai.yaml index ef77733e83..1c2418efab 100644 --- a/model/trace/gen-ai.yaml +++ b/model/trace/gen-ai.yaml @@ -36,9 +36,9 @@ groups: fine-tuned custom model, the value should have a more specific name than the base model that's been fine-tuned. - ref: gen_ai.response.finish_reasons requirement_level: recommended - - ref: gen_ai.usage.prompt_tokens + - ref: gen_ai.usage.input_tokens requirement_level: recommended - - ref: gen_ai.usage.completion_tokens + - ref: gen_ai.usage.output_tokens requirement_level: recommended events: - gen_ai.content.prompt diff --git a/schema-next.yaml b/schema-next.yaml index f87f1059d9..43e1afbef4 100644 --- a/schema-next.yaml +++ b/schema-next.yaml @@ -11,6 +11,11 @@ versions: messaging.rocketmq.client_group: messaging.consumer.group.name messaging.evenhubs.consumer.group: messaging.consumer.group.name message.servicebus.destination.subscription_name: messaging.destination.subscription.name + # TODO + - rename_attributes: + attribute_map: + gen_ai.usage.completion_tokens: gen_ai.usage.output_tokens + gen_ai.usage.prompt_tokens: gen_ai.usage.input_tokens spans: changes: # https://github.com/open-telemetry/semantic-conventions/pull/1002 From 0b2fdd8ef64dc5cdc451e9a0f81d0c5eefd32741 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 2 Jul 2024 13:29:16 -0700 Subject: [PATCH 2/5] changelog --- .chloggen/1200.yaml | 6 ++++++ schema-next.yaml | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) create mode 100644 .chloggen/1200.yaml diff --git a/.chloggen/1200.yaml b/.chloggen/1200.yaml new file mode 100644 index 0000000000..9d40d16961 --- /dev/null +++ b/.chloggen/1200.yaml @@ -0,0 +1,6 @@ +change_type: enhancement +component: gen_ai +note: > + Rename `gen_ai.usage.prompt_tokens` to `gen_ai.usage.input_tokens` and `gen_ai.usage.completion_tokens` to `gen_ai.usage.output_tokens` + to align terminology between spans and metrics. +issues: [ 1200 ] diff --git a/schema-next.yaml b/schema-next.yaml index 43e1afbef4..b2d310bd77 100644 --- a/schema-next.yaml +++ b/schema-next.yaml @@ -11,7 +11,7 @@ versions: messaging.rocketmq.client_group: messaging.consumer.group.name messaging.evenhubs.consumer.group: messaging.consumer.group.name message.servicebus.destination.subscription_name: messaging.destination.subscription.name - # TODO + # https://github.com/open-telemetry/semantic-conventions/pull/1200 - rename_attributes: attribute_map: gen_ai.usage.completion_tokens: gen_ai.usage.output_tokens From 203aa757bb5bdcb2bd6f7683798fa225bf812dcb Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 2 Jul 2024 13:30:28 -0700 Subject: [PATCH 3/5] lint --- model/registry/deprecated/gen-ai.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/model/registry/deprecated/gen-ai.yaml b/model/registry/deprecated/gen-ai.yaml index 5d37bd3da8..da2b6e49d0 100644 --- a/model/registry/deprecated/gen-ai.yaml +++ b/model/registry/deprecated/gen-ai.yaml @@ -4,13 +4,13 @@ groups: brief: Describes deprecated `gen_ai` attributes. attributes: - id: gen_ai.usage.prompt_tokens - type: string + type: int stability: experimental deprecated: Replaced by `gen_ai.usage.input_tokens` attribute. brief: "Deprecated, use `gen_ai.usage.input_tokens` instead." examples: [42] - id: gen_ai.usage.completion_tokens - type: string + type: int stability: experimental deprecated: Replaced by `gen_ai.usage.output_tokens` attribute. brief: "Deprecated, use `gen_ai.usage.output_tokens` instead." From 0a86f640ccfb56cefedc3a83b975d4263369997c Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 2 Jul 2024 13:34:28 -0700 Subject: [PATCH 4/5] lint --- docs/attributes-registry/gen-ai.md | 8 ++++---- model/registry/deprecated/gen-ai.yaml | 1 - 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/attributes-registry/gen-ai.md b/docs/attributes-registry/gen-ai.md index e8b76d93cd..8f98971a84 100644 --- a/docs/attributes-registry/gen-ai.md +++ b/docs/attributes-registry/gen-ai.md @@ -61,7 +61,7 @@ For custom model, a custom friendly name SHOULD be used. If none of these option Describes deprecated `gen_ai` attributes. -| Attribute | Type | Description | Examples | Stability | -| -------------------------------- | ------ | ----------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | -| `gen_ai.usage.completion_tokens` | string | Deprecated, use `gen_ai.usage.output_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.output_tokens` attribute. | -| `gen_ai.usage.prompt_tokens` | string | Deprecated, use `gen_ai.usage.input_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.input_tokens` attribute. | +| Attribute | Type | Description | Examples | Stability | +| -------------------------------- | ---- | ----------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `gen_ai.usage.completion_tokens` | int | Deprecated, use `gen_ai.usage.output_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.output_tokens` attribute. | +| `gen_ai.usage.prompt_tokens` | int | Deprecated, use `gen_ai.usage.input_tokens` instead. | `42` | ![Deprecated](https://img.shields.io/badge/-deprecated-red)
Replaced by `gen_ai.usage.input_tokens` attribute. | diff --git a/model/registry/deprecated/gen-ai.yaml b/model/registry/deprecated/gen-ai.yaml index da2b6e49d0..ca6abd6732 100644 --- a/model/registry/deprecated/gen-ai.yaml +++ b/model/registry/deprecated/gen-ai.yaml @@ -15,4 +15,3 @@ groups: deprecated: Replaced by `gen_ai.usage.output_tokens` attribute. brief: "Deprecated, use `gen_ai.usage.output_tokens` instead." examples: [42] - From f216878b52ead732cd3bc05124b9e6f8ebac7c88 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Tue, 2 Jul 2024 20:24:34 -0700 Subject: [PATCH 5/5] review --- docs/attributes-registry/gen-ai.md | 2 +- docs/gen-ai/gen-ai-spans.md | 2 +- model/registry/gen-ai.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/attributes-registry/gen-ai.md b/docs/attributes-registry/gen-ai.md index 8f98971a84..753b0c35e1 100644 --- a/docs/attributes-registry/gen-ai.md +++ b/docs/attributes-registry/gen-ai.md @@ -31,7 +31,7 @@ This document defines the attributes used to describe telemetry in the context o | `gen_ai.response.model` | string | The name of the model that generated the response. | `gpt-4-0613` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `gen_ai.system` | string | The Generative AI product as identified by the client or server instrumentation. [3] | `openai` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `gen_ai.token.type` | string | The type of token being counted. | `input`; `output` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `gen_ai.usage.input_tokens` | int | The number of tokens used in the GenAI input or prompt. | `100` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `gen_ai.usage.input_tokens` | int | The number of tokens used in the GenAI input (prompt). | `100` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | `gen_ai.usage.output_tokens` | int | The number of tokens used in the GenAI response (completion). | `180` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | **[1]:** It's RECOMMENDED to format completions as JSON string matching [OpenAI messages format](https://platform.openai.com/docs/guides/text-generation) diff --git a/docs/gen-ai/gen-ai-spans.md b/docs/gen-ai/gen-ai-spans.md index a92b2c2a54..1615d18c48 100644 --- a/docs/gen-ai/gen-ai-spans.md +++ b/docs/gen-ai/gen-ai-spans.md @@ -57,7 +57,7 @@ These attributes track input data and metadata for a request to an GenAI model. | [`gen_ai.response.finish_reasons`](/docs/attributes-registry/gen-ai.md) | string[] | Array of reasons the model stopped generating tokens, corresponding to each generation received. | `["stop"]` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | [`gen_ai.response.id`](/docs/attributes-registry/gen-ai.md) | string | The unique identifier for the completion. | `chatcmpl-123` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the model that generated the response. [3] | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| [`gen_ai.usage.input_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI input or prompt. | `100` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| [`gen_ai.usage.input_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI input (prompt). | `100` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | | [`gen_ai.usage.output_tokens`](/docs/attributes-registry/gen-ai.md) | int | The number of tokens used in the GenAI response (completion). | `180` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | **[1]:** The name of the GenAI model a request is being made to. If the model is supplied by a vendor, then the value must be the exact name of the model requested. If the model is a fine-tuned custom model, the value should have a more specific name than the base model that's been fine-tuned. diff --git a/model/registry/gen-ai.yaml b/model/registry/gen-ai.yaml index db474db703..06b994bfc0 100644 --- a/model/registry/gen-ai.yaml +++ b/model/registry/gen-ai.yaml @@ -93,7 +93,7 @@ groups: - id: usage.input_tokens stability: experimental type: int - brief: The number of tokens used in the GenAI input or prompt. + brief: The number of tokens used in the GenAI input (prompt). examples: [100] - id: usage.output_tokens stability: experimental