Skip to content
This repository has been archived by the owner on Dec 6, 2024. It is now read-only.

Commit

Permalink
Apply suggestions from code review
Browse files Browse the repository at this point in the history
Co-authored-by: Armin Ruech <[email protected]>
  • Loading branch information
cyrille-leclerc and arminru authored Mar 31, 2022
1 parent 00a6996 commit f0fec03
Showing 1 changed file with 11 additions and 11 deletions.
22 changes: 11 additions & 11 deletions text/0000-support-elastic-common-schema-in-opentelemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,20 @@ Collaborators: Alolita Sharma (OTEL GC, AWS), Cyrille Le Clerc (Elastic), Daniel

## Introduction

This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into Otel Semantic Conventions.
This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into OTel Semantic Conventions.

## Proposed process to merge ECS in OpenTelemetry Semantic Conventions

We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions:
1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented merging ECS fields in OpenTelemetry Semantic Conventions,
1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented by merging ECS fields in OpenTelemetry Semantic Conventions,
2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of Otel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/text.html#match-only-text-field-type">match_only_text</a>, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#keyword-field-type">keyword</a> <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type">constant_keyword</a>, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#wildcard-field-type">wildcard</a>),
3. Actual merge of ECS fields in Otel Semantic Conventions.

Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we prefered to first validate the principle and, once consensus is reached, actually define the mapping.

## Motivation

Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals.
Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals.

Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTEL Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition.

Expand All @@ -31,9 +31,9 @@ We'd like to see different categories of structured logs being well-supported in

## Customer Motivation

Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor generated along with open source logs.
Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs.

Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL compatible observability products and services.
Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL-compatible observability products and services.

OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)).

Expand All @@ -52,14 +52,14 @@ The coverage of ECS is very broad including in depth support for logs, security,

ECS has the following guiding principles:

* ECS favors human readability in order to enable broader adoption,
* ECS favors human readability in order to enable broader adoption as many fields can be understood without having to read up their meaning in the reference,
* ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...),
* ECS doesn't differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes),
* ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTEL does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes,
* ECS groups fields in namespaces in order to:
* Offer consistency and readability,
* Enable reusability of namespaces in different contexts,
* For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces
* Enable extensibility adding fields to namespaces and adding new namespaces,
* Enable extensibility by adding fields to namespaces and adding new namespaces,
* Prevent field name conflicts
* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It's much broader than simple logging use cases.

Expand Down Expand Up @@ -136,8 +136,8 @@ Example of a Nginx Access Log entry structured with ECS

| Description | [OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition) | [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) |
|-------------|-------------|--------|
| Metadata shared by all the Log Messages / Spans of an application instance | Resource Attributes | ECS fields |
| Metadata specific to each Log Message / Span | Attributes | ECS Fields |
| Metadata shared by all the Log Messages / Spans / Metrics of an application instance | Resource Attributes | ECS fields |
| Metadata specific to each Log Message / Span / Metric data point | Attributes | ECS Fields |
| Message of log events | Body | [message field](https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-message) |
| Naming convention | Dotted names | Dotted names |
| Reusability of namespaces | Namespaces are intended to be composed | Namespaces are intended to be composed |
Expand Down Expand Up @@ -251,7 +251,7 @@ It would be useful to get interoperability between metrics collected by [officia

Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which doesn't enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts.

Prometheus' conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don't specify unified metric names.
Prometheus' conventions are restricted to the style of the name of the metrics (see [Prometheus Metric and label naming](https://prometheus.io/docs/practices/naming/)) but don't specify unified metric names.


## Other areas that need to be addressed by OTEL (the project)
Expand Down

0 comments on commit f0fec03

Please sign in to comment.