How does a caller figure out the trace id of a request?

[codefromthecrypt]

For debugging or (log) correlation reasons, a user might want to know what the trace id is of a request. Since X-B3 headers are request headers (not response headers), how does a user know which trace is related to their request?

[codefromthecrypt]

There's a number of ways to accomplish this.

Specify the X-B3-TraceId and X-B3-SpanId headers

In B3, the recipient of X-B3-TraceId and X-B3-SpanId resumes that span, even if they immediately create a child. If you want to know the trace identifiers, specify them!

Keep in mind that unless X-B3-Sampled: 1, the trace might be dropped due to sampling. If you are just interested in controlling ids, leave out the X-B3-Sampled header and let the next hop decide. If you need the specific trace to be sampled, set X-B3-Flags: 1, indicating it is a debug span requested to pass all sampling (even collection tier).

For example, here's zipkin's browser extension controlling trace identifiers https://github.com/openzipkin/zipkin-browser-extension/blob/master/js/attachBeforeSendHeadersListener.js#L15

Add a binary annotation corresponding to a application-defined request id

While B3 doesn't affect the backtrace (response headers), applications often do. For example, they might return x-myapp-requestid: xxxxxxx. Zipkin has key-value search capabilities based on "binary annotations" which is a fancy name for tags. When a tracer starts a Span associated with a B3 initiated trace, it can add the application request id as a binary annotation in the root span. Then, a user can search by the request id they know.

[jcarres-mdsol]

I'd say X-B3-TraceId is all a user may want and it can be just added to response headers. At least in the implementations I deal with it would be trivial as it is a middleware in the request/response stack

[nicmunroe]

We return X-B3-TraceId in response headers by default. Unfortunately there are some cases where a caller needs to know what the trace ID will be before it makes the call (i.e. in the case of fire & forget scenarios where the response cannot be inspected) - in those cases the caller sends in the B3 headers it wants to be used. So 👍 for @adriancole's description covering both cases.

[codefromthecrypt]

I didn't mention my opinion on returning X-B3-TraceId.. something I should have. First, the audience for this project are hopefully less than 50 developers of tracing libraries, not end-users.

Long story short is that I think it is very worthwhile to put docs around what tracer authors should know about when thinking B3, I'd be careful to propagate B3 directly to end users as this limits our ability to change in the future.

---

X-B3-TraceId isn't a great user-level id. In fact, I suspect we will want to change to a new propagation model in the future. The more users directly interact with X-B3-TraceId the tougher it is to ever change to a new propagation model.

specifying ids on the way out should ideally be done via a tool like an instrumented curl or swagger thing. That way, it can change easily (as that tooling is centralized and has a changelog). Getting users used to B3 like they are Content-Type might solidify something late in the game. We may wish to change to a more conscious approach of id provisioning in the future. Ex one that more explicitly understands the nature of request ids vs trace ids, like envoy https://lyft.github.io/envoy/docs/intro/arch_overview/tracing.html cc @mattklein123

your thoughts?

[codefromthecrypt]

ex this is a swagger decorator which allows power-users a way to initiate traces, using B3 without knowing how it works https://github.com/Yelp/swagger_zipkin by @prat0318

[mattklein123]

@adriancole I don't have a lot of context on this, but FYI @RomanDzhabarov did all of the tracing work in Envoy if you have any questions and some folks from LightStep might also be interested in this (@jmacd @djspoons @bensigelman).

[codefromthecrypt]

Ah cool just pinged you on account of commit volume to envoy (also assumed
some x-twitter folks might be implicated in envoy and may have known about
B3)

On 16 Sep 2016 11:15, "Matt Klein" [email protected] wrote:

@adriancole https://github.com/adriancole I don't have a lot of context
on this, but FYI @RomanDzhabarov https://github.com/RomanDzhabarov did
all of the tracing work in Envoy if you have any questions and some folks
from LightStep might also be interested in this (@jmacd
https://github.com/jmacd @djspoons https://github.com/djspoons
@bensigelman https://github.com/bensigelman).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#4 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAD61_TIRiYMPOe6N5LHlqaymTjU7jVTks5qqgm2gaJpZM4J8TkV
.

[codefromthecrypt]

here's an example of "option 2" which is used by jaeger by @yurishkuro jaegertracing/jaeger-client-java#53

[nicmunroe]

@adriancole could the B3 docs state that it's an application concern whether or not the trace ID is returned in the response? i.e. it's not part of the spec and could change at any time so callers should only ever use it as a nice-to-have debugging tool and not rely on it for anything mission critical?

I guess I'm not clear on why returning trace ID limits future flexibility or locks us into anything.

[codefromthecrypt]

@adriancole https://github.com/adriancole could the B3 docs state that
it's an application concern whether or not the trace ID is returned in the
response? i.e. it's not part of the spec and could change at any time so
callers should only ever use it as a nice-to-have debugging tool and not
rely on it for anything mission critical?

For those who are writing B3 explicitly as http middleware, it probably
doesn't seem to matter if we copy a header from here to there.

Remember, though, that B3 has never had scope for response, not in sampling
or anything else. B3 code in brave and opentracing is extracted to outbound
propagation libraries that only affects requests (there are generic
extractors here), and are reused across multiple middleware. Whatever we
document shouldn't be confusing or imply users should completely remodel
their abstractions to now include the backtrace.

IOTW, I'd want to be careful about separating B3 propagation very
explicitly from the response, or application/middleware practice or
convention. Remember that http is only one of the ways trace data can
propagate. In non-http propagation (like thrift, cassandra), these fields
are passed binary and have no "X-" prefix.

The part about me personally being uninterested in logging B3-Trace-Id as a
response header is mostly not wanting to answer questions about why
everyone doesn't do this to higher orders of people. I'm implicitly signed
up, right :) When we start documenting things that only one or two do, I
end up having to answer why. Then.. guess what happens when we define a
real middleware spec, similar to envoy.. I end up having to explain why the
response header is now x-request-id? Get me? Selfishness of my time aside,
I'm just saying we should think this through and export things very
carefully to larger orders of people. None of this limits what individual
libraries can do... I mean good specs are built from existing practice.
This practice has to come from somewhere, right? :)

anyway....

Want to have a go at drafting what that "current practice" for this might
look like? Here's some hints about log correlation (which should be a
separate issue if we start taking on generic middleware concerns). Ex in
logging, I don't think most use the header name, though what people do is
all over the map. Ex some call the log correlation field "traceId" (and
don't bother with the span id), some call it "zipkinTraceId".

[nicmunroe]

@adriancole totally understood on wanting to avoid situations that generate timewasting questions, especially on something that doesn't need to be (or shouldn't be) part of the spec. I support whatever you think is best on that front since you have way more experience with community interaction and the consequences of these kinds of documentation decisions.

I was mostly looking to figure out if I needed to change wingtips - the servlet filter helper class currently automatically sets the X-B3-TraceId header on the response. I could change it to return something like X-Wingtips-RequestId if that would be preferable.

[codefromthecrypt]

@adriancole https://github.com/adriancole totally understood on wanting
to avoid situations that generate timewasting questions, especially on
something that doesn't need to be (or shouldn't be) part of the spec. I
support whatever you think is best on that front since you have way
more experience with community interaction and the consequences of these
kinds of documentation decisions.

consequences usually are two ways, not just here, more generally I think:
perceived complexity (ex people see various half-implemented options and
think the project is a mess) or limitations on refactoring (hard to extract
request id middleware because of backwards compat).

Worse is being useless due to analysis paralysis. Ex there's a real need to
facilitate how to know the trace id, but we do nothing! That's one of the
reasons I wanted to enumerate things on this issue.. to see which parts we
could support generally, and which could be advisory.

I was mostly looking to figure out if I needed to change wingtips - the
servlet filter helper class currently automatically sets the X-B3-TraceId
header on the response. I could change it to return something like
X-Wingtips-RequestId if that would be preferable.

Personally, I would change it to something like "X-Wingtips-RequestId" or
similar because in this case, Wingtips is the edge and that which would add
the response header. There could be a future where people do the same thing
generically, and then there's a standard header everyone uses (B3, not B3,
etc), like "X-Trace.. (wait XTrace is a project).. errr you know what I
mean :P Anyway, "X-Wingtips-RequestId" or similar would be cool,
especially if it also ties to log correlation practice. In B3, when we
discuss adding a request id to the response, we could hint at use cases for
it, and examples like Wingtips.

[basvanbeek]

I think the concept of a correlation-id to tie requests together with tracers and logs is not the scope of Zipkin and all about the scope of the consumer / frameworks handling transport details. See openzipkin/openzipkin.github.io#48 (comment)

Long story short. It is the concern of the ones building a platform to make sure they originate a correlation identifier at the source of the request and properly handle the propagation of it, so any type of instrumentation backend can record this correlation id with the hooks they provide (binary annotations, contextual logging key-value pairs, etc.). With this we don't introduce scope creep and reliance on something we shouldn't be promising in the first place as it exceeds the domain of tracing.