Messaging consumers flag to force starting a new trace instead of child span

[jeqo]

Feature:
Adding a messaging consumer flag to opt-out from continuing a trace, and create a new one.

Rational
Messaging pipelines, where a message producer is fan-out to (potentially) multiple consumers, we should allow users to tell which of these consumers should form part of the same trace, and which should start a new trace (probably linking these via tags).

Example Scenario
We have producer1->topic1 and consumers topic1->consumer1, topic1->consumer2 and topic1->consumer3; where consumer 1 and 2 are part of the same "pipeline" as producer 1, but consumer3->topic2 where other consumers subscribe to.

In this scenario, if we have a messaging consumer flag, we could say on consumer 3:

MessagingTracing.create(...).forceNewTraceOnReceive(true)

Leading to 2 traces for each message:

trace 1: producer1:send ... consumer1:receive .. consumer2:receive ...

trace 2: consumer3:receive consumer3:send ... consumerN:receive ...

Where trace 2 consumer:receive span can be tagged with a "parentTraceId" linking to trace 1.

[jorgheymans]

Any thoughts whether the flag could be applied selectively rather than having it global for all tracing covered by the MessagingTracing instance ? Is there a case for wanting to enable it for a particular topic only, and not for others ? Imagine a stream application that runs within one ecosystem of services (meaning it should not 'break' traces using this flag), but also feeds from another 'outside' application that already has gigantic traces it's not interested in and will thus apply this flag for.

[jeqo]

@jorgheymans wouldn't be possible in the scenario you mentioned that consumer for the feeds from outside application is configured with a different MessagingTracing instance? Would this solve your use-case?

[jorgheymans]

@jeqo that came to my mind as well, not sure how common this is though having multiple tracing instances around. Also how would this work when using sleuth, I'm guessing there will be quite a bit of config plumbing to do. Not saying we should cater for this use case now, but let's keep it in mind.

[jeqo]

@timtebeek would you mind to take a look to this proposed feature and how it matches with sleuth?

[timtebeek]

Hi @jeqo ; Was away for a few days, sorry for the slow response.. Not sure if I have much to add here; Right now the Sleuth instrumentation creates a single instance of KafkaStreamsTracing, and I don't see it able to handle more than one instance right now (although I've not explored that in depth), based on how the beans are wired together.

While I'd welcome the option to always start a new trace, my guess is such a flag would have to be applied globally for the entire kafka streams application topology, or similarly for non streams applications. Don't suppose that's much of a problem though, just something to point out based on the conversation above.

So, if we make the assumption that trace continuation can only be disabled for an entire Sleuth-using-application, then such a flag would be of value only if your application does a fan out and nothing else, I suppose, since any subsequent operations within the same application would also forgo span continuation. Is that correct?

At work we have a process that does a 1:10.000 fan out, which could use such a flag to prevent a sampled initial record leading to tens of thousands fan out spans reporting to Zipkin. Although it would require trimming down the service such that any subsequent operations happen within a different streams topology, just so those operations are not reported as single span traces.

[jeqo]

Thanks @timtebeek

As this flag will only be used on consumer side to continue a trace it will be useful with or without fan-out.

Now that use mention Kafka Streams applications I realize there might be multiple consumers on them (e.g. through() operators), and we can be interested only on starting a new trace at the entry point, aligned with @jorgheymans comments. I see the value of setting certain topics for this now.