tracing: Transition to OpenTelemetry from OpenTracing and OpenCensus

[moderation]

Title: Plan to transition to OpenTelemetry

Description: Envoy currently supports OpenTracing (OTr) and OpenCensus (OC) [0]. In May 2019 it was announced that these projects were merging into the new OpenTelemetry (OTel) project [1]. The original plan was to have the legacy project repos moved to read-only by the end of 2019. That hasn't happened but according to the OTel maintainers they are aiming for beta releases in March [2].

Should Envoy:

• keep OTr and OC and add OTel (eventually deprecating the legacy projects)?
• replace OTr and OC with OTel?

#9955 is planning on adding to the OC capability. The OC service / agent repo says its in maintenance mode and points to OTel/opentelemetry-collector

Relevant Links:
0 : config.trace.v3.Tracing.Http
1 : OpenTracing, OpenCensus Merge into a Single New Project, OpenTelemetry
2 : OpenTelemetry Monthly Update: January 2020

[yvespp]

Probably also depends on the C++ otel library so it can be used in envoy.
https://github.com/open-telemetry/opentelemetry-cpp is still in active development and has no releases yet.

[moderation]

AWS adds observability metrics to the OpenTelemetry C++ library

[moderation]

Getting closer - RC1 - https://opensource.googleblog.com/2020/10/opentelemetrys-first-release-candidates.html

[moderation]

Announced at re:invent December 2020 - AWS Distro for OpenTelemetry - https://aws-otel.github.io/

[gramidt]

As a member of the OpenTelemetry community, I thought I would share that the OpenTelemetry Tracing specification 1.0.1 has been released.

Long term support includes:

• API: 3 year support guarantee
• Plugin Interfaces: 1 year support guarantee
• Constructors: 1 year support guarantee

Announcement blog: https://medium.com/opentelemetry/opentelemetry-specification-v1-0-0-tracing-edition-72dd08936978

Link to specifications: https://github.com/open-telemetry/opentelemetry-specification

[gramidt]

@mattklein123 - Thoughts on next steps for this? I'm happy to coordinate with members of the OpenTelemetry community to help with the implementation.

[mattklein123]

@gramidt in general the tracing work tends to be driven by vendors and those with a vested interest. We are more than happy to see this work done so please go for it once resources are found! Thank you.

[gramidt]

Sounds like a plan, @mattklein123! Thank you for the prompt response.

[CoderPoet]

Is there any progress on this issue?

[gramidt]

Hi @CoderPoet -

I'm not aware of any progress here. Is this an immediate priority for you/your team?

[rakyll]

@gramidt Did you kick off any conversations to coordinate any work for this? We are interested to see OTLP support from Envoy at least for tracing data so we can position the Otel collector in the export path.

[rakyll]

One more thing...

There are two main areas when it comes to OpenTelemetry support:

1. Exporting OTLP spans from Envoy
2. Instrumenting Envoy with OpenTelemetry and removing OpenTracing

We can tackle (1) and (2) independently. (2) is possible without having to reinstrument Envoy by linking the OpenTelemetry <-> OpenTracing bridge. Eventually, we can get to (2) to remove the OpenTracing dependency. I think Envoy users can benefit from (1) immediately, and we should prioritize it.

[gramidt]

@rakyll - I have had some conversations both internally and externally in the community, but no progress has been made that I am aware of. A recent message from an employee of Adobe says that they may have someone who would be interested in making the contribution for exporting OTLP spans from Envoy.

[danielgblanco]

Has it be considered to break up (2) above into an OpenTracing shim vs a full native OpenTelemetry implementation? One aspect that would solve is context propagation in environments that are currently only using TraceContext (or composite propagators without B3 being one of them), without having to wait for a full OpenTelemetry instrumentation. One could export spans via OTLP (1), but if context propagation still depends on B3 headers, which are becoming less used in new environments, the full value of tracing will not be achieved.

[Tenaria]

Hi @gramidt,

Has there been any update on this in regards to resourcing the work? We also have a similar interest on the topic for the same reason as @rakyll and was wondering if there is any planned roadmap / expected timeline at this stage.

Thanks!

[gramidt]

@Tenaria - Sadly, I have not heard of any progress on this.

[dio]

@gramidt sorry for a naive question: do we intend to use https://github.com/open-telemetry/opentelemetry-cpp? Last time I check it has a similar approach with the current OC impl. By exporting OTLP spans only, is it converting OC Spans to OTLP's?

[AlexanderEllis]

Hi @gramidt, I'd be interested in taking on this work!

I've been discussing the potential approach with Harvey (@htuch), and it looks like it may make sense to add a Tracer that uses the OpenTelemetry protos and C++ API. The only potential issue is the stability of the C++ SDK, and depending on how stable it is (I see there was a 1.0.0 release recently), it may make sense to use Envoy's gRPC service instead of the SDK for exporting (similar to what @itamarkam did for the OpenTelemetry logger extension in #15105).

Either way, still interested in tackling this this :)

[lakamsani]

Hi, folks thanks for the Open Telemetry (OTel) support. It works and we are using it. As documented here we are using something like this:

                tracing:
                  provider:
                    name: envoy.tracers.opentelemetry
                    typed_config:
                      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                      grpc_service:
                        envoy_grpc:
                          cluster_name: opentelemetry_collector
                        timeout: 0.250s
                      service_name: my-envoy

You may be aware that the OTel project is suggesting a few OTEL_ prefixed env vars to drive the config like OTEL_SERVICE_NAME for service_name and OTEL_EXPORTER_OTLP_ENDPOINT for the collector endpoint. We confirmed they work for some custom go and java services where we are using OTel SDKs.

Might be a worthwhile enhancement for Envoy along this OTel support journey along with sending metrics along to the same OTel collector endpoint. thanks 🙏🏽

[DanTulovsky]

Hi.. is it possible to add additional configuration parameters to the tracer. For example "max_batch_size" which, seems to default to 5 and is much too low.

thank you!

[AlexanderEllis]

@DanTulovsky yup that's configurable via the tracing.opentelemetry.min_flush_spans setting: https://github.com/envoyproxy/envoy/blob/main/source/extensions/tracers/opentelemetry/tracer.cc#L148 👍

[DanTulovsky]

@AlexanderEllis But how do I set that inside envoy given: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/trace/v3/opentelemetry.proto.html

?

[holycheater]

There's a problem mentioned in #25347

Is it possible to decouple trace extraction/propagation from actual reporting?
Because now switching to opentelemetry trace reporting breaks propagation between services which still use uber-trace-id or x-b3- headers.

[gregoryfranklin]

Its a shame envoy does not use the opentelemetry-cpp SDK, that has a lot of nice features like being able to choose the propagator. Although I can understand why it was implemented the way it was.

I've been playing around with the OpenTracing Shim in opentelemtry-cpp to create an OpenTelemetry tracer that works with the dynamic_ot tracer extension in envoy. This allows me to use all the SDK features (such as choosing a propagator).

If I ever get this working I will post a link.

[dceravigupta]

@esigo any update on HTTP exporter? Are you still working on it? Thanks!

[moderation]

OpenCensus repos will be archived on July 31 2023 - census-instrumentation/opencensus-proto@1664cc9?short_path=b335630#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5. Definitely time to deprecate the OpenCensus extension

[phlax]

cc @kyessenov related to recent conversations about this

[kyessenov]

Thanks, we are well aware of it and trying to move Istio away from OpenCensus.
Reference istio/istio#44743.

[phlax]

wrt opentracing this has a new urgency

we landed code to use the cmake/ninja toolchains provided by rules_foreign_cc in #27211

this broke a couple of (non-hermetic) builds that expect to use the host cmake (cf #27243 )

in the case of opentracing im wondering if its worth the effort fixing a dodgy/unmaintained build from 6 years ago, or whether the better fix is just to get rid

[phlax]

for ref the failing build #27244 (cc @keith )

[Oberon00]

For opentracing, the dynamic_ot configuration is not even deprecated https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/trace/v3/dynamic_ot.proto

I'd really urge you to follow your usual deprecation policy for this feature https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/deprecation#how-are-configuration-deprecations-handled

We currently use an approach that auto-rewrites envoy configuration files if Dynatrace-monitoring is installed and since there was no timeline yet from which time on the feature becomes deprecated, we haven't been able to add an EOL version. We have been relying on the general deprecation timeline to be made aware ahead of time so that we can gracefully de-support the feature. Others may rely on this as well.

[phlax]

I'd really urge you to follow your usual deprecation polic ... We currently use an approach that auto-rewrites envoy configuration files in Dynatrace and since there was no timeline yet from which time on the feature becomes deprecated, we haven't been able to add an EOL version.

ok - this is a fair point - perhaps we do need to stick to the policy here - the truth is this should have been deprecated some time ago as this has been on the cards for quite a while, and the removal discussed here and variously elsewhere

in this case i would argue a few things - one is that we have a responsibility to not include unmaintained software - which i think is the case here

if we go ahead and remove as suggested in #27246 then there is a fair amount of time to make any updates (next planned release is 2023/07/14)

the other option is we fix the build (cf #27244 ) - i have tried pretty hard to do that - it fails on a single test (asan) and afaict is not trivial to workaround, or desirable to disable what is being tested - the fix may well be trivial but beyond my cmake knowledge

i would be very open to reviewing a PR that fixes the build - which is currently broken due to non-hermetic cmake issues (only not triggered currently in our ci for that same reason)

if we can get that working then i have no objection to carrying this lib a bit longer

[Oberon00]

I can check if I can get some time scheduled for this build fix investigation.

If not, and you can't get it fixed, another option would be to remove the feature code but keep accepting the configuration with a warning and simply do nothing with it.

[Oberon00]

To not leave the last comment hanging in the air, a brief update/summary of what happened: The build problems have been resolved (#27400), the OpenTracing deprecation topic now has a dedicated issue: #27401

[joaopgrassi]

Hi all, just wanted to circle back on the topic of Envoy adopting the opentelemetry-cpp SDK. Is that something the community see as realistic happening at some point? Or is the long term plan to keep the existing tracing things as is and build upon it?

@gregoryfranklin sorry for the ping, but since you mentioned that you were looking into it, I wonder if you have any luck in getting something working?

We have now a PR to add the OTLP HTTP exporter which is awesome, but there are many other features that we won't have still, so I wonder if we shouldn't really think in adopting the OTel c++ SDK.

[npuichigo]

Kindly ask if there's any recent update?

[joaopgrassi]

Hi @npuichigo not an update on the whole transition, but worth nothing we added some OTel features to Envoy:

• Export traces via HTTP.

• Can enrich the spans with resource attributes using the environment resource detector.

• Base for adding custom samplers is there as well, with the AlwaysOnSampler added as example