LLM to GenAI and other clean ups

open-telemetry · May 28, 2024 · 9e601af · 9e601af
1 parent 805cad3
commit 9e601af
Show file tree

Hide file tree

Showing 11 changed files with 45 additions and 194 deletions.
diff --git a/docs/attributes-registry/gen-ai.md b/docs/attributes-registry/gen-ai.md
@@ -12,26 +12,27 @@ This document defines the attributes used to describe telemetry in the context o
 
 | Attribute                        | Type     | Description                                                                                      | Examples                                                                | Stability                                                        |
 | -------------------------------- | -------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ---------------------------------------------------------------- |
-| `gen_ai.completion`              | string   | The full response received from the LLM. [1]                                                     | `[{'role': 'assistant', 'content': 'The capital of France is Paris.'}]` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.completion`              | string   | The full response received from the GenAI model. [1]                                             | `[{'role': 'assistant', 'content': 'The capital of France is Paris.'}]` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | `gen_ai.operation.name`          | string   | The name of the operation being performed.                                                       | `chat`; `completion`                                                    | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.prompt`                  | string   | The full prompt sent to an LLM. [2]                                                              | `[{'role': 'user', 'content': 'What is the capital of France?'}]`       | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.request.max_tokens`      | int      | The maximum number of tokens the LLM generates for a request.                                    | `100`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.request.model`           | string   | The name of the LLM a request is being made to.                                                  | `gpt-4`                                                                 | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.request.temperature`     | double   | The temperature setting for the LLM request.                                                     | `0.0`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.request.top_p`           | double   | The top_p sampling setting for the LLM request.                                                  | `1.0`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.prompt`                  | string   | The full prompt sent to GenAI model. [2]                                                         | `[{'role': 'user', 'content': 'What is the capital of France?'}]`       | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.request.max_tokens`      | int      | The maximum number of tokens the model generates for a request.                                  | `100`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.request.model`           | string   | The name of the Generative AI model a request is being made to.                                  | `gpt-4`                                                                 | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.request.temperature`     | double   | The temperature setting for the GenAI request.                                                   | `0.0`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.request.top_p`           | double   | The top_p sampling setting for the GenAI request.                                                | `1.0`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | `gen_ai.response.finish_reasons` | string[] | Array of reasons the model stopped generating tokens, corresponding to each generation received. | `stop`                                                                  | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | `gen_ai.response.id`             | string   | The unique identifier for the completion.                                                        | `chatcmpl-123`                                                          | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.response.model`          | string   | The name of the LLM a response was generated from.                                               | `gpt-4-0613`                                                            | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.response.model`          | string   | The name of the model that generated the response.                                               | `gpt-4-0613`                                                            | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | `gen_ai.system`                  | string   | The Generative AI product as identified by the client instrumentation. [3]                       | `openai`                                                                | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | `gen_ai.token.type`              | string   | The type of token being counted.                                                                 | `input`; `output`                                                       | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.usage.completion_tokens` | int      | The number of tokens used in the LLM response (completion).                                      | `180`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| `gen_ai.usage.prompt_tokens`     | int      | The number of tokens used in the LLM prompt.                                                     | `100`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.usage.completion_tokens` | int      | The number of tokens used in the GenAI response (completion).                                    | `180`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| `gen_ai.usage.prompt_tokens`     | int      | The number of tokens used in the GenAI input or prompt.                                          | `100`                                                                   | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 
 **[1]:** It's RECOMMENDED to format completions as JSON string matching [OpenAI messages format](https://platform.openai.com/docs/guides/text-generation)
 
 **[2]:** It's RECOMMENDED to format prompts as JSON string matching [OpenAI messages format](https://platform.openai.com/docs/guides/text-generation)
 
 **[3]:** The actual GenAI product may differ from the one identified by the client. For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` is set to `openai` based on the instrumentation's best knowledge.
+For custom model, a custom friendly name SHOULD be used. If none of these options apply, the instrumentation the `gen_ai.system` SHOULD be set to `_OTHER`.
 
 `gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.
 

diff --git a/docs/gen-ai/README.md b/docs/gen-ai/README.md
@@ -14,16 +14,9 @@ The semantic conventions for GenAI and LLM are currently in development.
 We encourage instrumentation libraries and telemetry consumers developers to
 use the conventions in limited non-critical workloads and share the feedback
 
-This document defines semantic conventions for the following kind of Generative AI systems:
-
-* LLMs
-
 Semantic conventions for Generative AI operations are defined for the following signals:
 
 * [Metrics](gen-ai-metrics.md): Semantic Conventions for Generative AI operations - *metrics*.
+* [Spans](gen-ai-spans.md): Semantic Conventions for Generative AI requests - *spans*.
 
-Semantic conventions for LLM operations are defined for the following signals:
-
-* [LLM Spans](llm-spans.md): Semantic Conventions for LLM requests - *spans*.
-
-[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.26.0/specification/document-status.md
+[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.33.0/specification/document-status.md
diff --git a/docs/gen-ai/gen-ai-metrics.md b/docs/gen-ai/gen-ai-metrics.md
@@ -68,14 +68,15 @@ This metric SHOULD be specified with [ExplicitBucketBoundaries] of [1, 4, 16, 64
 | Attribute  | Type | Description  | Examples  | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability |
 |---|---|---|---|---|---|
 | [`gen_ai.operation.name`](/docs/attributes-registry/gen-ai.md) | string | The name of the operation being performed. | `chat`; `completion` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| [`gen_ai.request.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the LLM a request is being made to. | `gpt-4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| [`gen_ai.request.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the Generative AI model a request is being made to. | `gpt-4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client instrumentation. [1] | `openai` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`gen_ai.token.type`](/docs/attributes-registry/gen-ai.md) | string | The type of token being counted. | `input`; `output` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`server.port`](/docs/attributes-registry/server.md) | int | Server port number. [2] | `80`; `8080`; `443` | `Conditionally Required` If `sever.address` is set. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |
-| [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the LLM a response was generated from. | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the model that generated the response. | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`server.address`](/docs/attributes-registry/server.md) | string | Server domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name. [3] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |
 
 **[1]:** The actual GenAI product may differ from the one identified by the client. For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` is set to `openai` based on the instrumentation's best knowledge.
+For custom model, a custom friendly name SHOULD be used. If none of these options apply, the instrumentation the `gen_ai.system` SHOULD be set to `_OTHER`.
 
 **[2]:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available.
 
@@ -137,14 +138,15 @@ This metric SHOULD be specified with [ExplicitBucketBoundaries] of [ 0.01, 0.02,
 | Attribute  | Type | Description  | Examples  | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability |
 |---|---|---|---|---|---|
 | [`gen_ai.operation.name`](/docs/attributes-registry/gen-ai.md) | string | The name of the operation being performed. | `chat`; `completion` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
-| [`gen_ai.request.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the LLM a request is being made to. | `gpt-4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| [`gen_ai.request.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the Generative AI model a request is being made to. | `gpt-4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client instrumentation. [1] | `openai` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`error.type`](/docs/attributes-registry/error.md) | string | Describes a class of error the operation ended with. [2] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | `Conditionally Required` if the operation ended in an error | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |
 | [`server.port`](/docs/attributes-registry/server.md) | int | Server port number. [3] | `80`; `8080`; `443` | `Conditionally Required` If `sever.address` is set. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |
-| [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the LLM a response was generated from. | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
+| [`gen_ai.response.model`](/docs/attributes-registry/gen-ai.md) | string | The name of the model that generated the response. | `gpt-4-0613` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) |
 | [`server.address`](/docs/attributes-registry/server.md) | string | Server domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name. [4] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |
 
 **[1]:** The actual GenAI product may differ from the one identified by the client. For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` is set to `openai` based on the instrumentation's best knowledge.
+For custom model, a custom friendly name SHOULD be used. If none of these options apply, the instrumentation the `gen_ai.system` SHOULD be set to `_OTHER`.
 
 **[2]:** The `error.type` SHOULD match the error code returned by the Generative AI provider or the client library,
 the canonical name of exception that occurred, or another low-cardinality error identifier.
@@ -176,7 +178,7 @@ Instrumentations SHOULD document the list of errors they report.
 <!-- END AUTOGENERATED TEXT -->
 <!-- endsemconv -->
 
-[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md
+[ExplicitBucketBoundaries]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.33.0/specification/metrics/api.md#instrument-advisory-parameters
+[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.33.0/specification/document-status.md
 [MetricRequired]: /docs/general/metric-requirement-level.md#required
 [MetricRecommended]: /docs/general/metric-requirement-level.md#recommended
-[ExplicitBucketBoundaries]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.31.0/specification/metrics/api.md#instrument-advisory-parameters