Support instrumenting Spring Sleuth

[Diagoras]

While Datadog supports instrumenting Spring Boot applications, it doesn't take into account the detailed trace data generated by the Spring Cloud Sleuth module. One way to deal with this would be for Sleuth to report traces to Datadog's built-in trace API - something I've created an issue for already. However, it might be nicer for instrumented applications to have the agent automatically recognize Sleuth traces, and might result in duplicate trace data (one set from Sleuth over the API, the other set coming directly from the tracing agent).

What do you think? A worthwhile goal, or one better solved by the issue I linked above?

[tylerbenson]

I created https://github.com/tylerbenson/zipkin-datadog-reporter as an experimental side project, but haven't gotten around to writing docs or publishing to maven. We are also experimenting with collecting zipkin spans "natively". This second option is interesting because it wouldn't require code changes by users and could work with any language implementation.

[gdubya]

We are currently investigating using Sleuth to monitor our apps and would be interested in reporting these to datadog instead of implementing our own solution. What's the best way to send the traces to DD right now?

[tylerbenson]

@gdubya Any particular reason you don't want to use dd-java-agent? Otherwise there's a fork of my repo mentioned above that's actually been published: https://github.com/smartupio/zipkin-datadog-reporter

[gdubya]

No, I suppose that is the best way once we take our other monitoring requirements into consideration (jmx reporting, etc). Thanks for the feedback!

[renannprado]

@tylerbenson @gdubya I'm going through the quest of configuring sleuth+datadog right now.
The reason I want do it with sleuth is that if we change from datadog to something else, sleuth is abstracting a lot of things such as the annotations you can use to create spans.
I believe I'm about to succeed, but if someone already has a how-to, it would be nice.

[tylerbenson]

@renannprado our javaagent actually has instrumentation for Sleuth's annotation: https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/trace-annotation/src/main/java/datadog/trace/instrumentation/trace_annotation/TraceAnnotationsInstrumentation.java#L41

[renannprado]

@tylerbenson does datadog agent propagate tracing with B3 format when using, for example, gRPC? That's my main restriction actually.

[tylerbenson]

@renannprado currently we don't support B3, but we do support propagation over gRPC with dd-java-agent. Is that not working for you?

[renannprado]

I does work, but I need to propagate with B3 so make the distributed tracing integration vendor neutral.

[aantono]

I think #717 would be helpful to enable configuring such integrations.

[tylerbenson]

FYI: We are adding support for B3 in #756.

[masoncj]

Just came here to say that we spent a little while trying to get Spring Sleuth/Brave to integrate well with Datadog's java agent, and eventually gave up and fell back to using open tracing direct to Datadog. It seems like Spring Sleuth is pretty tightly coupled to Brave so substituting other Tracers seems challenging. The datadog reporter approach would seem to work but would require forgoing other features in the datadog java agent (submitting trace data directly to datadog's monitoring agent).

It would be really nice if there were some instructions around about how to configure spring sleuth to work well with datadog.

[tvcsantos]

We started using datadog and currently in our spring-boot applications we were using spring-boot-sleuth with the brave tracer, to automatically add in our logs traceId and spanId propagatting them to other services/applications.

What I can see is that by adding the following depencies:

<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-java-agent</artifactId>
    <version>${datadog.version}</version>
</dependency>
<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-trace-api</artifactId>
    <version>${datadog.version}</version>
</dependency>

and by setting DD_LOGS_INJECTION: true we can now have APM logs/trace correlations but traceId/spanId from sleuth do not match the ones sent to datadog.

So what should be the step to have correctly configured spring-boot-sleuth brave with datadog. Can someone help on this?

[marcingrzejszczak]

@masoncj with Sleuth 3.x we're abstracting tracers as such. E.g. we have a separate project https://github.com/spring-cloud-incubator/spring-cloud-sleuth-otel/ where OpenTelemetry is the tracer we're using. Maybe you can give it another shot?

[AndersClausen]

FWIW @masoncj , we're currently using what @marcingrzejszczak is referring to and it's working really well.

[avnerstr]

Hi @AndersClausen

I followed the instructions that appears here: https://spring-cloud-incubator.github.io/spring-cloud-sleuth-otel/docs/current/reference/html/howto.html#howto
and it still didn't work with Datadog.

What else do I need to add?

[AndersClausen]

Hi @avnerstr Have you got a repo you can share? There really isn't much you need to do anymore. You do have an instance of the Otel Collector running right? What does you POM and application.yml files look like?

[elue]

any update on this issue?

[bm1549]

@elue we've made some improvements to our Spring handling in #4770

Can you try the latest version of Spring (6.1.0) and Datadog (1.24.1) and let me know if the issue still exists?

[marcingrzejszczak]

Just FYI Spring Cloud Sleuth 3.1.x is feature complete and you should migrate to Micrometer Tracing https://github.com/spring-cloud/spring-cloud-sleuth/#-important-

[KafkaProServerless]

Upvoting this.

[elue]

@bm1549 we moved to B3 headers, so no longer uses this