[RFC] Integrate Opentelemetry

[weyert]

Describe the feature you'd like to request

I would like to see to Traces of the executed build similar to profiling json that can be exported instead of it going to a JSON file it will create Trace spans which can be exported via the OTLP protocol of Opentelemetry to any supported vendor platform (e.g. otel-collector etc)

Describe the solution you'd like

I would like to be able to use the CNCF standard for tracing the Opentelemetry standard (www.opentelemetry.io). It's has great support by the popular tracing vendors, such as New Relic, Datadog, or Dynatrace, Splunk.

Describe alternatives you've considered

Converter of profiler.json to Opentelemetry traces via an external tool

[gsoltis]

@weyert run_state.go includes support for outputting this information in a format for the chrome profiler. If we want more profiling output options, maybe that would be the integration point?

[weyert]

Cool, I will have a look 🤓

[mehulkar]

@arlyon can you take this one? I think your work is applicable here, but I'm not 100% sure if it matches what @weyert is looking for

[arlyon]

The good news is I have been working on this for use with the new rust codepath. I will update this issue when it is available (under the experimental-rust-codepath flag for now)

[yehonatanz]

Any news regarding this? This feature could be a game changer for us when it comes to build time analysis.
It would be even more amazing if it would somehow play well with buildkit's own OTEL support, but that's mostly nice to have I guess.

[arlyon]

I do have a small update. I was hacking on this last night and was able to get it mostly working with one caveat: the tracing library does not allow runtime configuration. I have opened an issue here: tokio-rs/tracing-opentelemetry#121

But, what it does mean is that I have context propagation working both from buildx to turbo and from turbo to other downstream tools (using the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and TRACEPARENT env vars). It's pretty cool :) So if the tasks turbo runs for you supports opentelemetry it should all appear in one unified trace.

Blue / yellow is buildkit, brown is turbo.

Here is a trace you can drop into jaeger to have a poke around.

1a0d49160a2e0b4264b7a12b7fc697ff.json

[arlyon]

Here is the command to use in your dockerfile. buildx sets up TRACEPARENT for you automatically

RUN unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT && /app/turbo run test --otlp-destination http://localhost:4317 --otlp-parent $TRACEPARENT

[arlyon]

https://github.com/vercel/turbo/tree/arlyon/otel

[yehonatanz]

This is so awesome!

[jaridmargolin]

@arlyon - This looks awesome!

Are there any plans to merge this into main? OTEL traces make it very easy to see where we were able to run things in parallel and where we had blocking tasks.