Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make otel spans for RR clients more spec compliant #26694

Merged
merged 1 commit into from
Jul 25, 2022

Conversation

kdubb
Copy link
Contributor

@kdubb kdubb commented Jul 13, 2022

The OpenTelementry Spec is pretty clear about not using the URI as the default span name due to its high cardinality.

Instrumentation MUST NOT default to using URI path as span name…

Previously that's exactly what the span name extractor was doing.

Understanding that there are a number of issues and PRs related to reducing the cardinality of the client span names. After working in a live environment with telemetry enabled I've made some changes to make the span names compliant and easier to disambiguate in UIs like Jaeger.

  1. This new format never uses the raw URI, as dictated by the spec.
  2. The format always starts with HTTP followed by the method, when available.
  3. If the route template is available it is appended last.

Examples:

HTTP
HTTP GET
HTTP GET /test

Always prepending the HTTP <method> disambiguates the client and server spans (which might share the same exact route template) in the UI. In the example below you can see how the edge service uses a client to call the data service. The client and server have the exact same route template but with the prefixing it easily distinguishable.

Screen Shot 2022-07-12 at 9 11 06 PM

This example might seem a little redundant since it's in the hierarchical format but we find it much easier to read. Also, in other flat formats the client and server are quite clear without drilling down.

@kdubb kdubb requested a review from radcortez July 13, 2022 04:00
@quarkus-bot
Copy link

quarkus-bot bot commented Jul 13, 2022

/cc @radcortez

@quarkus-bot

This comment has been minimized.

@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from 2f50b08 to bcdfc3f Compare July 13, 2022 04:34
@quarkus-bot

This comment has been minimized.

@geoand
Copy link
Contributor

geoand commented Jul 13, 2022

The test failures seems related to the change

@kdubb
Copy link
Contributor Author

kdubb commented Jul 13, 2022

Yep. This is me juggling the separate changes and forgetting to rebuild and test both modules.

@kdubb
Copy link
Contributor Author

kdubb commented Jul 13, 2022

@geoand I'm just realizing how closely linked this PR and #26693 are.

@kdubb
Copy link
Contributor Author

kdubb commented Jul 13, 2022

This new output format is uses the route template which is only available after the other PR.

@geoand
Copy link
Contributor

geoand commented Jul 13, 2022

@geoand I'm just realizing how closely linked this PR and #26693 are.

Yeah, I wonder if there should be a single PR (with multiple commits and tests)

@kdubb
Copy link
Contributor Author

kdubb commented Jul 13, 2022

At the very least, using this name format, the the handler order is tested. If the route template becomes unavailable again due to a code change it should show up in the OpenTelemetry tests.

Copy link
Member

@radcortez radcortez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests also needs to be fixed.

@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from bcdfc3f to fe31790 Compare July 14, 2022 17:53
@kdubb
Copy link
Contributor Author

kdubb commented Jul 14, 2022

The tests are fixed now that the other PR is merged.

@quarkus-bot

This comment has been minimized.

@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from fe31790 to 5d2dddb Compare July 14, 2022 23:35
@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from 5d2dddb to 56c4353 Compare July 21, 2022 16:34
@kdubb
Copy link
Contributor Author

kdubb commented Jul 21, 2022

@radcortez As discussed this removes the custom extractor and replaces it with the default HttpSpanNameExtractor

@radcortez
Copy link
Member

Thank you for the PR @kdubb!

@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from 69b90c7 to 1049658 Compare July 22, 2022 16:27
@kdubb kdubb added triage/waiting-for-ci Ready to merge when CI successfully finishes triage/backport? labels Jul 22, 2022
The OpenTelementry [Spec](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.10.0/specification/trace/semantic_conventions/http.md#name) is pretty clear about not using the URI as the default span name due to its high cardinality.

> Instrumentation MUST NOT default to using URI path as span name…

This PR creates span names that always include `HTTP` and then if available the method (e.g. `GET`) or `request`.
@kdubb kdubb force-pushed the fix/otel_rr_client_span_name branch from 1049658 to 3ce1ac2 Compare July 25, 2022 16:06
@gsmet gsmet merged commit a4ae3a1 into quarkusio:main Jul 25, 2022
@quarkus-bot quarkus-bot bot added this to the 2.12 - main milestone Jul 25, 2022
@quarkus-bot quarkus-bot bot removed the triage/waiting-for-ci Ready to merge when CI successfully finishes label Jul 25, 2022
@gsmet
Copy link
Member

gsmet commented Jul 25, 2022

@radcortez Kevin would like me to backport this one. I'm not sure if we want to change this behavior in a micro. But maybe it makes sense given the previous behavior was not the one we wanted?
Let me know.

@kdubb kdubb deleted the fix/otel_rr_client_span_name branch July 25, 2022 18:59
@radcortez
Copy link
Member

I think it is ok to backport.

@gsmet gsmet modified the milestones: 2.12 - main, 2.11.1.Final Jul 26, 2022
michalvavrik added a commit to michalvavrik/quarkus-test-suite that referenced this pull request Jul 27, 2022
Logic behind Rest Client span names has changed in quarkusio/quarkus#26694, this reflect changes and fixes related test failures in daily CI.
fedinskiy pushed a commit to quarkus-qe/quarkus-test-suite that referenced this pull request Jul 27, 2022
Logic behind Rest Client span names has changed in quarkusio/quarkus#26694, this reflect changes and fixes related test failures in daily CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants