Unable to debug OpenTelemetry traces // OpenTelemetry traces not being generated?

[kencieszykowski]

Hey everyone,

Trying to get OpenTelemetry traces ingested, and currently don't have a way to verify that Triton is actually sending any traces to our OTLP endpoint. We are running on Kubernetes v1.23.

Running 23.04 currently, and here is the startup command (this will likely get moved to a ConfigMap at some point):

tritonserver --model-repository=[GCS bucket] \
--disable-auto-complete-config \
--log-verbose=0 --allow-gpu-metrics=true \
--metrics-interval-ms=1 --trace-config mode=opentelemetry \
--trace-config level=TIMESTAMPS --trace-config rate=1 \
--trace-config opentelemetry,url="http://[OTLP Endpoint]:4318/v1/traces"

I've exec'd into the Triton pod and sent a sample trace JSON to the endpoint directly, which was ingested just fine-- so network connectivity and collector configuration seems to be properly set up.

The issue is that I'm just seeing nothing from Triton. I've combed through documentation, tried playing around with the Trace API extension, and at this point reaching the end of my rope with how to further debug or monitor OpenTelemetry traces.

It seems that local trace logging is only available for the triton trace configuration, but if anybody has any further insight I'm all ears.

Thank you!

[kencieszykowski]

So jumped the gun here-- if anybody has similar issues the fix is:

--trace-config opentelemetry,url=http://[OTLP Endpoint]:4318/v1/traces

Turns out there was a log line on the server about a malformed URL on the OTLP Spec that I missed. Oof.