Gateway 0.x/2.x Open Telemetry traces missing with otel collector 0.43.0+

[prasek]

Problem

otel/opentelemetry-collector v0.43.0 removed deprecated legacy path ("v1/trace") support for otlp http receiver

• Remove deprecated legacy path support for otlp http receiver open-telemetry/opentelemetry-collector#4720
• otlp receiver expects traces on “/v1/trace” but spec says “/v1/traces”. open-telemetry/opentelemetry-collector#1968

supergraph-demo with otel/opentelemetry-collector 0.43.0 shows error samples in zpages:

• http://localhost:55679/debug/tracez shows errors http.status_code=404, http.target=/v1/trace

Error Trace Spans

3 Requests

When                       Elapsed (sec)
----------------------------------------
2022/02/10-02:39:24.916957      .    77 trace_id: a8f6732587ff36e0e1b45c5c6e335322 span_id: 6aa3347a98d8cbbc
                                        Status{Code=Error, description=""}
                                        Attributes:{http.flavor=1.1, http.host=collector:55681, http.method=POST, http.request_content_length=5047, http.scheme=http, http.status_code=404, http.target=/v1/trace, http.wrote_bytes=19, net.host.name=collector, net.host.port=55681, net.peer.ip=172.22.0.2, net.peer.port=41562, net.transport=ip_tcp}
2022/02/10-02:39:32.619622      .    35 trace_id: 944f49a15fd55768af3df35c7f965a3f span_id: 67bb4fd243bc30ef
                                        Status{Code=Error, description=""}
                                        Attributes:{http.flavor=1.1, http.host=collector:55681, http.method=POST, http.request_content_length=30437, http.scheme=http, http.status_code=404, http.target=/v1/trace, http.wrote_bytes=19, net.host.name=collector, net.host.port=55681, net.peer.ip=172.22.0.6, net.peer.port=59220, net.transport=ip_tcp}
2022/02/10-02:39:35.647788      .    41 trace_id: 9e672ce2916c5e67929e3254dc8c3b08 span_id: c00e14f481d47c0d
                                        Status{Code=Error, description=""}
                                        Attributes:{http.flavor=1.1, http.host=collector:55681, http.method=POST, http.request_content_length=29390, http.scheme=http, http.status_code=404, http.target=/v1/trace, http.wrote_bytes=19, net.host.name=collector, net.host.port=55681, net.peer.ip=172.22.0.6, net.peer.port=59220, net.transport=ip_tcp}

TraceId means sampled request. TraceId means not sampled request.

Workaround

otel/opentelemetry-collector 0.42.0 is the last release that supports /v1/trace

• pin otel collector 0.42.0 for /v1/trace supergraph-demo#396
• pin otel collector 0.42.0 for /v1/trace supergraph-demo-fed2#44

Gateway 0.x Repro

• git clone [email protected]:apollographql/supergraph-demo.git
• bump otel/opentelemetry-collector to 0.43.0 or 0.44.0
• make docker-up-otel-collector
• make smoke
• open http://localhost:55679/debug/tracez
• see error traces with http.status_code=404, http.target=/v1/trace

Gateway 2.x Repro

• git clone [email protected]:apollographql/supergraph-demo-fed2.git
• bump otel/opentelemetry-collector to 0.43.0 or 0.44.0
• make docker-up-otel-collector
• make smoke
• open http://localhost:55679/debug/tracez
• see error traces with http.status_code=404, http.target=/v1/trace

Subgraphs

• subgraphs are also not reporting traces with otel/opentelemetry-collector 0.43+
• currently referenced https://www.npmjs.com/package/@opentelemetry/node lib is deprecated
• Federation Docs should reference current otel libs

[abernix]

My understanding would be that the collector would (now) be accepting on only the newer /v1/traces end-point (instead of the deprecated trace), but it's the exporter that would also have a definition of that (now, incorrect) /v1/trace or (correct) /v1/traces.

Could this be due to a dependency on the deprecated @opentelemetry/exporter-collector npm package which is a dependency of the supergraph-demo-opentelemetry package that you've published and that supergraph-demo uses here? It appears that package says that the default URL is v1/trace (which makes sense, since that used to be correct!). It's worth noting that it still does allow setting the url property on the collectorOptions that are passed to CollectorTraceExporter.

It appears that the current/maintained @opentelemetry/exporter-trace-otlp-http both allows specifying the endpoint URL and defaults to the /v1/traces endpoint.

Thoughts?

[abernix]

Separately, it is indeed true that some of the packages listed in the documentation are deprecated — like @opentelemetry/tracing.