Using OpenTelemetry Tracing
This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide distributed tracing for interactive web applications.
|
Note
|
|
Prerequisites
Architecture
In this guide, we create a straightforward REST application to demonstrate distributed tracing.
Solution
We recommend that you follow the instructions in the next sections and create the application step by step. However, you can skip right to the completed example.
Clone the Git repository: git clone {quickstarts-clone-url}, or download an {quickstarts-archive-url}[archive].
The solution is located in the opentelemetry-quickstart directory.
Creating the Maven project
First, we need a new project. Create a new project with the following command:
{includes}/devtools/create-app.adoc
This command generates the Maven project and imports the quarkus-opentelemetry extension,
which includes the default OpenTelemetry support,
and a gRPC span exporter for OTLP.
If you already have your Quarkus project configured, you can add the quarkus-opentelemetry extension
to your project by running the following command in your project base directory:
{includes}/devtools/extension-add.adoc
This will add the following to your build file:
<dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-opentelemetry</artifactId> </dependency>
implementation("io.quarkus:quarkus-opentelemetry")
Examine the Jakarta REST resource
Create a src/main/java/org/acme/opentelemetry/TracedResource.java file with the following content:
package org.acme.opentelemetry; import jakarta.ws.rs.GET; import jakarta.ws.rs.Path; import jakarta.ws.rs.Produces; import jakarta.ws.rs.core.MediaType; import org.jboss.logging.Logger; @Path("/hello") public class TracedResource { private static final Logger LOG = Logger.getLogger(TracedResource.class); @GET @Produces(MediaType.TEXT_PLAIN) public String hello() { LOG.info("hello"); return "hello"; } }
Notice that there is no tracing specific code included in the application. By default, requests sent to this endpoint will be traced without any required code changes.
Create the configuration
{includes}/opentelemetry-config.adoc
Run the application
First we need to start a system to visualise the OpenTelemetry data. We have 2 options:
-
Start an all-in-one Grafana OTel LGTM system for traces and metrics.
-
Jaeger system just for traces.
Grafana OTel LGTM option
-
Take a look at: Getting Started with Grafana-OTel-LGTM.
This features a Quarkus Dev service including a Grafana for visualizing data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides and OTel collector to receive the data.
Jaeger to see traces option
Configure and start the OpenTelemetry Collector to receive, process and export telemetry data to Jaeger that will display the captured traces.
|
Note
|
Jaeger-all-in-one includes the Jaeger agent, an OTel collector, and the query service/UI. You do not need to install a separated collector. You can directly send the trace data to Jaeger (after enabling OTLP receivers there, see e.g. this blog entry for details). |
Start the OpenTelemetry Collector and Jaeger system via the following docker-compose.yml file that you can launch via docker-compose up -d:
version: "2" services: # Jaeger jaeger-all-in-one: image: jaegertracing/all-in-one:latest ports: - "16686:16686" # Jaeger UI - "14268:14268" # Receive legacy OpenTracing traces, optional - "4317:4317" # OTLP gRPC receiver - "4318:4318" # OTLP HTTP receiver, not yet used by Quarkus, optional - "14250:14250" # Receive from external otel-collector, optional environment: - COLLECTOR_OTLP_ENABLED=true
You should remove the optional ports you don’t need them.
Start the application
Now we are ready to run our application. If using application.properties to configure the tracer:
or if configuring the OTLP gRPC endpoint via JVM arguments:
{includes}/devtools/dev.adoc :!dev-additional-parameters:
With the OpenTelemetry Collector, the Jaeger system and the application running, you can make a request to the provided endpoint:
$ curl http://localhost:8080/hello hello
When the first request has been submitted, you will be able to see the tracing information in the logs:
10:49:02 INFO traceId=, parentId=, spanId=, sampled= [io.quarkus] (main) Installed features: [cdi, opentelemetry, resteasy-client, resteasy, smallrye-context-propagation, vertx]
10:49:03 INFO traceId=17ceb8429b9f25b0b879fa1503259456, parentId=3125c8bee75b7ad6, spanId=58ce77c86dd23457, sampled=true [or.ac.op.TracedResource] (executor-thread-1) hello
10:49:03 INFO traceId=ad23acd6d9a4ed3d1de07866a52fa2df, parentId=, spanId=df13f5b45cf4d1e2, sampled=true [or.ac.op.TracedResource] (executor-thread-0) helloThen visit the Jaeger UI to see the tracing information.
Hit CTRL+C or type q to stop the application.
JDBC
The JDBC instrumentation will add a span for each JDBC queries done by your application, to enable it, add the following dependency to your build file:
<dependency> <groupId>io.opentelemetry.instrumentation</groupId> <artifactId>opentelemetry-jdbc</artifactId> </dependency>
implementation("io.opentelemetry.instrumentation:opentelemetry-jdbc")
As it uses a dedicated JDBC datasource wrapper, you must enable telemetry for your datasource:
# enable tracing quarkus.datasource.jdbc.telemetry=true # configure datasource quarkus.datasource.db-kind=postgresql quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/mydatabase
Additional configuration
Some use cases will require custom configuration of OpenTelemetry. These sections will outline what is necessary to properly configure it.
ID Generator
The OpenTelemetry extension will use by default a random ID Generator when creating the trace and span identifier.
Some vendor-specific protocols need a custom ID Generator, you can override the default one by creating a producer.
The OpenTelemetry extension will detect the IdGenerator CDI bean and will use it when configuring the tracer producer.
@Singleton public class CustomConfiguration { /** Creates a custom IdGenerator for OpenTelemetry */ @Produces @Singleton public IdGenerator idGenerator() { return AwsXrayIdGenerator.getInstance(); } }
Propagators
OpenTelemetry propagates cross-cutting concerns through propagators that will share an underlying Context for storing state and accessing
data across the lifespan of a distributed transaction.
By default, the OpenTelemetry extension enables the W3C Trace Context and the W3C Baggage
propagators, you can however choose any of the supported OpenTelemetry propagators by setting the propagators config that is described in the OpenTelemetry Configuration Reference.
Additional Propagators
-
The
b3,b3multi,jaegerandottracepropagators will need the trace-propagators extension to be added as a dependency to your project.
<dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-extension-trace-propagators</artifactId> </dependency>
implementation("io.opentelemetry:opentelemetry-extension-trace-propagators")
-
The
xraypropagator will need the aws extension to be added as a dependency to your project.
<dependency> <groupId>io.opentelemetry.contrib</groupId> <artifactId>opentelemetry-aws-xray-propagator</artifactId> </dependency>
implementation("io.opentelemetry.contrib:opentelemetry-aws-xray-propagator")
Customise Propagator
To customise the propagation header you can implement the TextMapPropagatorCustomizer interface. This can be used, as an example, to restrict propagation of OpenTelemetry trace headers and prevent potentially sensitive data to be sent to third party systems.
/** * /** * Meant to be implemented by a CDI bean that provides arbitrary customization for the TextMapPropagator * that are to be registered with OpenTelemetry */ public interface TextMapPropagatorCustomizer { TextMapPropagator customize(Context context); interface Context { TextMapPropagator propagator(); ConfigProperties otelConfigProperties(); } }
Resource
See the main OpenTelemetry Guide resources section.
End User attributes
When enabled, Quarkus adds OpenTelemetry End User attributes as Span attributes. Before you enable this feature, verify that Quarkus Security extension is present and configured. More information about the Quarkus Security can be found in the Quarkus Security overview.
The attributes are only added when authentication has already happened on a best-efforts basis.
Whether the End User attributes are added as Span attributes depends on authentication and authorization configuration of your Quarkus application.
If you create custom Spans prior to the authentication, Quarkus cannot add the End User attributes to them.
Quarkus is only able to add the attributes to the Span that is current after the authentication has been finished.
Another important consideration regarding custom Spans is active CDI request context that is used to propagate Quarkus SecurityIdentity.
In principle, Quarkus is able to add the End User attributes when the CDI request context has been activated for you before the custom Spans are created.
quarkus.otel.traces.eusp.enabled=true (1) quarkus.http.auth.proactive=true (2)
-
Enable the End User Attributes feature so that the
SecurityIdentityprincipal and roles are added as Span attributes. The End User attributes are personally identifiable information, therefore make sure you want to export them before you enable this feature. -
Optionally enable proactive authentication. The best possible results are achieved when proactive authentication is enabled because the authentication happens sooner. A good way to determine whether proactive authentication should be enabled in your Quarkus application is to read the Quarkus Proactive authentication guide.
|
Important
|
This feature is not supported when a custom Jakarta REST SecurityContexts is used. |
Sampler
A sampler decides whether a trace should be discarded or forwarded, effectively managing noise and reducing overhead by limiting the number of collected traces sent to the collector.
Quarkus comes equipped with a built-in sampler, and you also have the option to create your custom sampler.
To use the built-in sampler, you can configure it by setting the desired sampler parameters as detailed in the OpenTelemetry Configuration Reference. As an example, you can configure the sampler to retain 50% of the traces:
# build time property only: quarkus.otel.traces.sampler=traceidratio # Runtime property: quarkus.otel.traces.sampler.arg=0.5
|
Tip
|
An interesting use case for the sampler is to activate and deactivate tracing export at runtime, according to this example: # build time property only:
quarkus.otel.traces.sampler=traceidratio
# On (default). All traces are exported:
quarkus.otel.traces.sampler.arg=1.0
# Off. No traces are exported:
quarkus.otel.traces.sampler.arg=0.0 |
If you need to use a custom sampler there are now 2 different ways:
Sampler CDI Producer
You can create a sampler CDI producer. The Quarkus OpenTelemetry extension will detect the Sampler CDI bean and will use it when configuring the Tracer.
@Singleton public class CustomConfiguration { /** Creates a custom sampler for OpenTelemetry */ @Produces @Singleton public Sampler sampler() { return JaegerRemoteSampler.builder() .setServiceName("my-service") .build(); } }
OTel Sampler SPI
This will use the SPI hooks available with the OTel Autoconfiguration. You can create a simple Sampler class:
public class CustomSPISampler implements Sampler { @Override public SamplingResult shouldSample(Context context, String s, String s1, SpanKind spanKind, Attributes attributes, List<LinkData> list) { // Do some sampling here return Sampler.alwaysOn().shouldSample(context, s, s1, spanKind, attributes, list); } @Override public String getDescription() { return "custom-spi-sampler-description"; } }
Then a Sampler Provider:
public class CustomSPISamplerProvider implements ConfigurableSamplerProvider { @Override public Sampler createSampler(ConfigProperties configProperties) { return new CustomSPISampler(); } @Override public String getName() { return "custom-spi-sampler"; } }
Write the SPI loader text file at resources/META-INF/services with name io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSamplerProvider containing the full qualified name of the CustomSPISamplerProvider class.
Then activate on the configuration:
quarkus.otel.traces.sampler=custom-spi-sampler
As you can see, CDI is much simpler to work with.
Additional instrumentation
Some Quarkus extensions will require additional code to ensure traces are propagated to subsequent execution. These sections will outline what is necessary to propagate traces across process boundaries.
The instrumentation documented in this section has been tested with Quarkus and works in both standard and native mode.
CDI
Annotating a method in any CDI aware bean with the io.opentelemetry.instrumentation.annotations.WithSpan
annotation will create a new Span and establish any required relationships with the current Trace context.
Annotating a method in any CDI aware bean with the io.opentelemetry.instrumentation.annotations.AddingSpanAttributes will not create a new span but will add annotated method parameters to attributes in the current span.
If a method is annotated by mistake with @AddingSpanAttributes and @WithSpan annotations, the @WithSpan annotation will take precedence.
Method parameters can be annotated with the io.opentelemetry.instrumentation.annotations.SpanAttribute annotation to
indicate which method parameters should be part of the span. The parameter name can be customized as well.
Example:
@ApplicationScoped class SpanBean { @WithSpan void span() { } @WithSpan("name") void spanName() { } @WithSpan(kind = SERVER) void spanKind() { } @WithSpan void spanArgs(@SpanAttribute(value = "arg") String arg) { } @AddingSpanAttributes void addArgumentToExistingSpan(@SpanAttribute(value = "arg") String arg) { } }
Available OpenTelemetry CDI injections
As per MicroProfile Telemetry Tracing specification, Quarkus supports the CDI injections of the following classes:
-
io.opentelemetry.api.OpenTelemetry -
io.opentelemetry.api.trace.Tracer -
io.opentelemetry.api.trace.Span -
io.opentelemetry.api.baggage.Baggage
You can inject these classes in any CDI enabled bean. For instance, the Tracer is particularly useful to start custom spans:
@Inject Tracer tracer; ... public void tracedWork() { Span span = tracer.spanBuilder("My custom span") .setAttribute("attr", "attr.value") .setParent(Context.current().with(Span.current())) .setSpanKind(SpanKind.INTERNAL) .startSpan(); // traced work span.end(); }
Quarkus Messaging - Kafka
When using the Quarkus Messaging extension for Kafka, we are able to propagate the span into the Kafka Record with:
TracingMetadata tm = TracingMetadata.withPrevious(Context.current()); Message out = Message.of(...).withMetadata(tm);
The above creates a TracingMetadata object we can add to the Message being produced,
which retrieves the OpenTelemetry Context to extract the current span for propagation.
Quarkus Security Events
Quarkus supports exporting of the Security events as OpenTelemetry Span events.
quarkus.otel.security-events.enabled=true (1)
-
Export Quarkus Security events as OpenTelemetry Span events.
Exporters
See the main OpenTelemetry Guide exporters section.
Quarkus core extensions instrumented with OpenTelemetry tracing
-
-
AMQP 1.0
-
RabbitMQ
-
Kafka
-
Pulsar
-
-
quarkus-vertx(http requests)
Disable parts of the automatic tracing
Automatic tracing instrumentation parts can be disabled by setting quarkus.otel.instrument.* properties to false.
Examples:
quarkus.otel.instrument.grpc=false quarkus.otel.instrument.messaging=false quarkus.otel.instrument.resteasy-client=false quarkus.otel.instrument.rest=false quarkus.otel.instrument.resteasy=false
Disable specific REST endpoints
There are two ways to disable tracing for a specific REST endpoint.
You can use the @io.quarkus.opentelemetry.runtime.tracing.Traceless (or simply @Traceless) annotation to disable tracing for a specific endpoint.
Examples:
@Traceless annotation on a class
@Path("/health") public class PingResource { @Path("/ping") public String ping() { return "pong"; } }
When the @Traceless annotation is placed on a class, all methods annotated with @Path will be excluded from tracing.
@Traceless annotation on a method
@Path("/trace") @Traceless public class TraceResource { @Path("/no") @GET @Traceless public String noTrace() { return "no"; } @Path("/yes") @GET public String withTrace() { return "yes"; } }
In the example above, only GET /trace/yes will be included in tracing.
Disable using configuration
If you do not want to modify the source code, you can use your application.properties to disable a specific endpoint through the quarkus.otel.traces.suppress-application-uris property.
Example:
quarkus.otel.traces.suppress-application-uris=trace,ping,people*This configuration will:
-
Disable tracing for the
/traceURI; -
Disable tracing for the
/pingURI; -
Disable tracing for the
/peopleURI and all other URIs under it, e.g.,/people/1,/people/1/cars.
| Note | If you are using |
OpenTelemetry Configuration Reference
See the main OpenTelemetry Guide configuration reference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation says:
I do not believe you can disable tracing for specific REST endpoints based on paths, because you cannot identify REST resource methods with path. Path to resource method is
one-to-manyrelation, but docs ATM says absolutely nothing about that.I am worried about this annotation approach:
@GETmethod with certain path? Won't this will also exclude@PUTand any other HTTP method? Same must be true for different consumed/produced types etc. if I have 10 different endpoints with same path and I put this one one endpoint, is it expected that all the endpoints are not traced?@Path("/hello/{id}") String method()and I have another method@Path("/hello/more-specific") completelyDifferentMethod()then both endpoints will be excluded even though I only placed this on one method with less specific path?@GETetc. but this feature depends on@Pathannotation, so here I cannot disable endpointget():@Pathon parent class endpoint or interface method without@Path, I'll get exception message containingPlease ensure that the class is properly annotated with @Path annotation..@Path, what happens if I put this on the REST client? Or better, if I put this on REST endpoints, this will not exclude traces for the client as well?I suggest that someone who makes living working on Jakarta REST impl. like @geoand reviews this PR. Maybe it's alright, I wouldn't know.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is absolutely true for JAX-RS / Jakarta REST. When I look at
@Traceless, I expect it's matching to behave the same way as@PermissionsAllowedfor example - this PR does not seem to do that at allThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe that
@Tracelessshould be removed completely as it's behavior is wrong. Allowing users to configure the path to not be traced makes sense, but with the way things are now,@Tracelessis very misleading and get users into a lot of trouble.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to make sure to properly acknowledge @mcruzdev for the work done here and @michalvavrik for raising this issue!
It's amazing to have such a great community that works so effectively on continuously improving Quarkus in so many ways!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to make sure to properly acknowledge @mcruzdev for the work done here and @michalvavrik for raising this issue!
It's amazing to have such a great community that works so effectively on continuously improving Quarkus in so many ways!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for catching this @michalvavrik. I really appreciate your reviews; I’ve been learning a lot.
We can remove the
@Tracelessannotation and rely solely on the configuration for now until we have a better implementation through annotations.@geoand can I send a pull request for removing it? From what I’ve seen, this hasn’t been released yet, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc: @brunobat
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes please do, this indeed has not been released yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was on a conference yesterday. I've replied in the new issue.