Modeling the body of a Semantic Convention

[MSNev]

What are you trying to achieve?

With the definition of Events storing their payload into the Log Body as defined in Semantic Conventions and as part of the Event Api there is a need to identify "how" the payload is described and encoded so that implementers and downstream consumers have a deterministic method of defining, decoding, processing and possible validating the content.

Therefore, we need to have a consistent way to describe the Log body.

There are 2 levels of definition that need to be considered, this issue will only focus on the Data Modeling of the body and not the transport encoding.

• Data Model: How is the body described in semantic conventions from a data model perspective for yaml, code generation and documentation
• Transport Encoding: How are the values encoded (with deterministic context) in the body of the Log
  • As this issue along could be quite large, I will assume the default position is as defined by Logs and the OTLP protocol as explicitly identifying the encoding does not necessarily need to be part of the data model.
  • And I would suggested that encoding should be discussed as part of a separate issue as it is a complex topic that will require a lot of discussion and agreement.

As the body is defined as an <any> we need to define how the body will be described in yaml and ultimately encoded for transportation.

Data Model

The body of a Log is defined as a value of type <any> which means that it can be a complex type that needs to be described in a consistent manner.

The following are some possible yaml representations for the body of a Log describing the expected content of the contents that will be included. These is based on the assumption that the body is a single value that can be described as a map<string, any>, array<any>, <scalar value> or a <byte array>.

Basic Modeling Rules

• Every body definition is unique only to the encompassing Semantic Convention that is declaring it.
• There will be no id assigned to the body definition, as it cannot be shared with other Data Models.
• Different body definitions may define the same or similar structure, but they will be considered to be unique to the encompassing Semantic Convention.

Extensions to theses basic modeling rules will be defined for each of the possible body definitions and can be extended in future releases of the specification if required.

map< string, any >

Additional Modeling Rules

• A body definition may have multiple field definitions.
• Each field definition within the body definition is unique to the encompassing body definition and may include an id that is unique to the body definition only.
• Any "fields" defined within the body definition are not global and are not available outside of their declared body definition. ie. They are not part of the global semantic conventions attributes.
• Fields within different body definitions may re-use the same id as they are unique to the encompassing body definition only, are not global and should not be considered as the same field definition.
• Any id defined within the body definition SHOULD not repeated withing the same fields definition.

body:
  fields:
    - id: <id>
      type: <type>
      brief: <brief>
      note: <note>
      examples: [<examples>]
    - id: <id>
      type: <type>
      brief: <brief>
      note: <note>
      examples: [<examples>]

Where fields is a list of field definitions where each field has an id that represents the key of the map and is described in the same manner as an Attribute with the biggest difference that the described fields are NOT added to the global semantic conventions attributes but are instead associated with the body of the encompassing Log.

Examples of using fields in a full semantic convention definition can be found in the Build tools draft PR to add log_event support

• yaml: Simple single field with generated markdown
• yaml: Simple single field with attributes with generated markdown
• yaml: Multiple fields with constraibts with generated markdown

This is the primary proposal for the event working group to support defining the body of an event in a consistent manner, so that it can be used for code generation, documentation and validation.

Open Questions

• Should Nested fields be allowed? ie. map<string, <any>> within the fields definition.
• Should the initial definition be that a map<string, <any>>, should the <any> be limited to not support nesting (map<string, <any>>) or do we allow the values to be encoded based on the identified "standard".
  • If nesting is allowed, there should also be a recommended defined "depth" for this nested.

array< any >

For completeness to describe the support OTLP protocol, it is possible to have an array of values as the body of a Log. This would be a list of values that are of type <any>.

body:
  array:
    brief: <brief>
    note: <note>
    examples: [<examples>]

Where array represents that there will be an unspecified list of values of type <any>, while it would contain the detail fields of an Attribute (brief, note, examples, etc) it would NOT provide an id. As the
array would assume to be numeric index based. Specific example will be required to identify if there are any specific "fields" that are expected to be present in the array. Like min/max length, min/max values, etc.

Open Question

• How or do we want to define the types of values that may be present in the array, along with additional attributes (Like min/max length, min/max values, etc).

scalar value

Also For completeness to describe the support OTLP protocol, it is possible to have an array of values as the body of a Log. This would be a list of values that are of type <any>.

body: 
  value:
    type: <type>
    brief: <brief>
    note: <note>
    examples: [<examples>]

Where <value> is described in the same manner as an Attribute and as with the array it would not include the id (key).

byte array

Also For completeness to describe the support OTLP protocol, it is possible to have an array of values as the body of a Log. This would be a list of values that are of type <any>.

This could also be rolled into the value definition as a type of binary but it may be better to have it as a separate definition to make it clear that the value is a binary blob.

body:
  binary:
    brief: <brief>
    note: <note>
    examples: [<examples>]

Where binary is described in the same manner as an Attribute and as with the array it would not include the id (key).

[lmolkova]

What drives the desire to describe full structure in yaml?
Given that you can put literally anything there, I don't see how flat structure described in fields could work to express something like

  "foo" : {"bar": [1,2,3,4,5], "baz":{ ......}}

There are big projects like json-schema that describe arbitrary schemas, can we reuse what they offer?

What would be broken if we did this:

body: 
  schema: link-to-schema goes here, e.g. https://support.google.com/analytics/answer/9216061#page_view
  brief: <brief>
  note: <note>
  examples: [<examples>]

then schema can be external to otel or page_view can be defined in otel, but not in the yaml. It can be as simple as manually written MD table.

I feel this would be the least controversial, easy to implement and use solution that can also be evolved to support stricter schema definitions.

I wonder if it'd be enough as a first step.

[lmolkova]

or even

body: 
  brief: <brief>. link-to-schema goes here, e.g. https://support.google.com/analytics/answer/9216061#page_view
  note: <note>
  examples: [<examples>]

[patrickhousley]

Personally, I would like the body to be represented in yaml. Having the body represented in yaml means that types will be generated and IDEs can use those to auto-complete what we would expect to be in the body of an event we define. If the body is just described in markdown, there is really nothing there "enforcing" the contents of the body for our defined events.

For the representation of body, I think we should keep it simple and stick with a collection of key/value pairs. I understand that this is not going to please everyone but it keeps our implementation simple in lieu of "body" being four possible data types and backends having to check the type for every event.

I understand that having body as map< string, any > is potentially wordy when using JSON transport. However, that should be a separate conversation as there are many possible solutions to how we transport the data.

Should Recursive fields be allowed? ie. map<string, > within the fields definition.

There are possible performance impacts if an event is excessively deep (nesting) or wide (flattened number of attributes). However, I think this needs to fall on the consumer of the API to control. If the consumer creates an event that has a degree of nesting/breadth that it does cause a performance issue, it must fall on them to fix that and the API should not try to catch/prevent this performance issue.

We can recommend a max depth/breadth but should not force it in code. With a recommendation, we can question any future PRs for defined events that exceed the recommendation.

[lmolkova]

To clarify my previous comment:

• I think ability to express body structure is good
• I don't think the ability to describe it the yaml is essential to make progress on the semconv
• If we started simple (free-form body definition in the MD), it would allow to
  • start adding event semconv with free form definition
  • work on structured yaml, codegen in parallel witout blocking any spec efforts on it

[MSNev]

I've just Updated the description of the issue to address and clarify some questions from the Event WG today, the changes are

• Removed / replaced the wording of "recursion" to "nested" / "nesting" as that is the intent of the description
• Changed the open question around "nesting" to be a "recommended" defined depth and not a mandated one
• Add some links to full examples of using the body definition for some additional context to what this issue is describing for yaml and the generated markdown. I don't currently have any relevant code generation in the linked Draft PR yet, but one of the intents of the code to be generated from the yaml is to generated interfaces to provide consistency of the field names used to avoid typos etc.
• Called out that the proposed array, scalar and binary are possible paths forward and are provided for completeness for all of the types supported by OTLP.

The intent of this issue is so that we can identify a path forward, and not necessarily to describe every possible option and enforce that as the Semantic Convention for yaml for all types.

As long as we can agree that we can define an optional body definition for some types (like an event) and we define it in a fashion to support the current requirements (defining fields specifically) without blocking possible future extensions (like array, scalar, binary, schema, etc).

Apart from the auto generation as already described, the main reason for raising this (and the (now linked) example draft build-tools PR) is that our previous attempts (years ago) to create simple free-form markdown definitions to describe the event payload (the initial use case) was previously rejected (and PR's where blocked).

While free-form markdown describing the "fields" is a possible option, it would still leave gaps for the code generation / validation, and we should really define a structure / guidance on what that "free-form" markdown should follow (IMHO).

[jsuereth]

A few thoughts:

Every body definition is unique only to the encompassing Semantic Convention that is declaring it.

Should this be "unique to the event.id" ?

There will be no id assigned to the body definition, as it cannot be shared with other Data Models.

Don't we need event.id to distinguish event types and understand WHICH event the body is describing?

For completeness to describe the support OTLP protocol, it is possible to have an array of values as the body of a Log. This would be a list of values that are of type <any>.

This is an oddity here. We, theoretically, disallow heterogeneous lists. Personally, I'd prefer typed lists here (with an explicit list[any] if we ever allow that).

[jsuereth]

I'd like to push back a little on

body: 
  value:
    type: <type>
    brief: <brief>
    note: <note>
    examples: [<examples>]
body:
  fields:

vs.

body:
  type: map
  brief:
  notes:
  fields:

That is - should we represent "Any" directly in body and be able to annotate or notes that?

Looking in the code, I understand thing will need to be disjoint, but I also don't think we have a good answer for nested maps and types.

I'd rather introduce one modelling concept for AnyValue rather than two and have to re-model structure again in body fields.

[lmolkova]

+1 for

body:
  type: map
  brief:
  notes:
  fields:

I assume we'd rarely have anything except map in the type

we can always do type: string or type: string[] (without fields).

Things get complicated the moment we get to complex types in arrays or maps either way (with extra value | fields | array layer or with type : something).

[MSNev]

Possible combinations for supporting "Any" value, following the examples / suggestions above.

  - id: single.field.primitive.or.array.body
    body:
      brief: ... for the body / field ...
      note: ... for the body / field ...
      type: boolean | boolean[] | int | int[] | double | double[] | string | string[]

   # Extending to support "map" (fields)
  - id: nested.map.of.primitives.or.array.body
    body:
      brief: ... for the body (any value) ...
      note: ... for the body (any value) ...
      type: map | map[]
      id: map_name???
      fields:
        - id: field1
          type: boolean | boolean[] | int | int[] | double | double[] | string | string[] | any_value
          brief: ... for the field ...
          examples: ... for the field ...
        - id: nested_map
          brief: ... for the field ...
          examples: ... for the field ...
          type: map
          fields:
            - id: nested_field1
              type: boolean | boolean[] | int | int[] | double | double[] | string | string[] | any_value
              brief: ... for the field ...
              examples: ... for the field ...
            - id: nested_field2
              type: boolean | boolean[] | int | int[] | double | double[] | string | string[] | any_value
              brief: ... for the field ...
              examples: ... for the field ...
            - id: nested_enum_name1
              type:
                allow_custom_values: true
                members:
                  - id: feature1
                    value: "f1"
                  - id: feature2
                    value: "f2"
            - id: nested_map_array
              type: map[]
              fields:
                - id: nested_map_array_field1
                  type: boolean | boolean[] | int | int[] | double | double[] | string | string[] | any_value
                  brief: ... for the field ...
                  examples: ... for the field ...
                - id: nested_map_array_field2
                  type: boolean | boolean[] | int | int[] | double | double[] | string | string[] | any_value
                  brief: ... for the field ...
                  examples: ... for the field ...

[lmolkova]

Implemented in weaver 0.10.0