LLM common metrics for Generative AI

[drewby]

Fixes #811

Changes

This adds initial metric definitions to the current set of gen_ai semantic conventions. These initial two metrics (gen_ai.usage.tokens and gen_ai.request.duration) are a minimal set to get started, and more can be added with future PRs.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[cartermp]

A question/observation:

Should we go through and just use "Gen AI Model" instead of "LLM" throughout the content of this document? Naming attributes genai.* but then having the descriptions talk about an LLM feels a little inconsistent to me now.

I think we're already at the point where the same model supports multi-modal inputs and outputs. Consider the following in Claude's API reference:

Starting with Claude 3 models, you can also send image content blocks:

{"role": "user", "content": [
  {
    "type": "image",
    "source": {
      "type": "base64",
      "media_type": "image/jpeg",
      "data": "/9j/4AAQSkZJRg...",
    }
  },
  {"type": "text", "text": "What is in this image?"}
]}

And with OpenAI it's a little less straightforward, but still possible:

An array of content parts with a defined type, each can be of type `text` or `image_url` when passing in images. You can pass multiple images by adding multiple `image_url` content parts. Image input is only supported when using the `gpt-4-visual-preview` model.

It's orthogonal to this PR, but maybe it's good to start here and not "limit" ourselves by using the LLM terminology, since it's usually associated with just text interpretation and generation?

[drewby]

A question/observation:

Should we go through and just use "Gen AI Model" instead of "LLM" throughout the content of this document? Naming attributes genai.* but then having the descriptions talk about an LLM feels a little inconsistent to me now.

I think we're already at the point where the same model supports multi-modal inputs and outputs. Consider the following in Claude's API reference:

Starting with Claude 3 models, you can also send image content blocks:

{"role": "user", "content": [
  {
    "type": "image",
    "source": {
      "type": "base64",
      "media_type": "image/jpeg",
      "data": "/9j/4AAQSkZJRg...",
    }
  },
  {"type": "text", "text": "What is in this image?"}
]}

And with OpenAI it's a little less straightforward, but still possible:

An array of content parts with a defined type, each can be of type `text` or `image_url` when passing in images. You can pass multiple images by adding multiple `image_url` content parts. Image input is only supported when using the `gpt-4-visual-preview` model.

It's orthogonal to this PR, but maybe it's good to start here and not "limit" ourselves by using the LLM terminology, since it's usually associated with just text interpretation and generation?

I agree both addressing this soon and doing it separate from this PR. I've updated anything I could that does not impact Spans yet (keeping this as the metrics PR). Let's create another PR to update the other references to LLM.

[lmolkova]

LGTM! Left a few minor-ish comments

[drewby]

LGTM! Left a few minor-ish comments

Thanks! All resolved.