Establish a trace context in a non-http context.

[jstafford5380]

Is your feature request related to a problem? Please describe.
Tracing seems dependent on an HTTP Context. When I am performing a function in response to a message coming in off of pubsub for example, I cannot establish an http context for the "request" nor would I want to, however the trace information may be available on the bus message so I should be able to re-establish the trace context and continue it. This results in broken traces once a transaction/span hits a service bus of any kind.

Describe the solution you'd like
I think ideally, the Activity object would be in place of (or in tandem with) the TraceHeaderContext. Activities are created whenever an entrypoint is engaged. For example when an HTTP request begins (incoming) or when the message from the bus is dispatched to a handler.

As luck would have it, the Activity also supports the W3C trace context format, making it easy to establish that. Since it would appear that this SDK doesn't yet support that either, we can use the tracestate to propogate the X-Cloud-Trace header and its value. Possibly knock out two enhancements in one. An adapter may be warranted.

Describe alternatives you've considered
I've considered creating a different scoped service. Sort of a property bag that would be accessible through DI but this wouldn't be consistent with how traces are establish when we actually DO have an HTTP context. It would be better to have a unified method, which I think lies in the Activity.

[jskeet]

Tracing itself isn't dependent on an HTTP context. Our ASP.NET Core part is, for natural reasons.

I suspect that the real answer is going to be integration with OpenTelemetry - to be honest, I hope that we can provide a good logging format to stdout that will avoid needing a logging API, and OpenTelemetry integration for tracing, allowing us to avoid a "heavy" integration at all.

Amanda may well have more information.

[jstafford5380]

@jskeet I agree that OpenTelemetry is probably a better fit, but I'm having a hard time figuring out where we're at with that. We're currently settled on the W3C trace context format and a few blogs suggest Google has (or is) implementing it and it is already available in some areas. I'm trying to find a solution that allows us to trace and log using W3C Trace Context to keep them linked. I agree that I'd rather just emit this stuff to stdout. I guess the greatest challenge right now is establish the trace context outside of an HTTP context.

We do have a support contract btw (we're a well known enterprise migrating from Azure to GCP, just now kicking off), and this is a bit of a sticking point. I wanted to gauge interest here first and I can open it inside of cloud console.

[jskeet]

I suspect you can do everything you need already with what's in the ASP.NET Core Diagnostics package, or with the "raw" tracing/logging APIs (which can certainly be used directly). Amanda may have more information. We haven't been prioritizing OpenTelemetry so far, partly as it's still in beta (and we're a small team), but I would hope we'll set aside some time at some point. (I appreciate that doesn't help you right now.)

[jstafford5380]

@jskeet Oh that's good information. This is a huge talking point with us. We were under the impression that we wouldn't have to use proprietary cloud trace headers as we build out other systems that are not hosted in GCP. We have a team that's trying to refine some logging in other non-cloud systems but pipe them through cloud trace, using that w3c format so we can trace full transactions across cloud and non cloud systems if that makes sense. If we still have to use X-Cloud-Trace-Context while OpenTelemetry integration is being developed, that would be good to know -- we could still continue as planned but utilize the tracestate to keep the cloud trace headers around.

Am I understanding this right? OpenTelemetry would be a different package that still sinks to stackdriver/cloud trace? Or is it something else completely?

Is there a roadmap somewhere by chance?

[jskeet]

I would expect OpenTelemetry integration to be via a different package, yes. (I suspect there'd be a package which is just a Cloud Trace sink, and relies on OpenTelemetry itself for all the integration.) We haven't had the time to look deeply at OpenTelemetry yet.

No roadmap for all this I'm afraid, more due to being busy with other work than anything else.

If you could give some more information about your scenario (are you using ASP.NET Core at all, or should we be talking about other contexts?) I suspect @amanda-tarafa will be able to help you more concretely than I can.

[amanda-tarafa]

@jstafford5380 I'll try to answer/comment on all your points, but please come back if I've missed something:

On OpenTelemetry: What have stopped us from even starting to look at OpenTelemetry integration is basically this notice on the .NET OpenTelemetry repo. It's not just that the library is in beta, but also that they don't guarantee to follow a specific version of the spec and are making emphasis on not even attempting to be backwards compatible from release to release. That means we'd be playing catch right now, and as @jskeet said we are a small team.

On the W3C format: Some of GCP's infrastructure already supports traceparent while still supporting x-cloud-trace-context. I'm sorry the best information I have right now for you is this GoogleCloudPlatform/esp-v2#252 (comment) from another Googler. I will try to get more shareable details internally. From this issue I can't be sure what GCP infrastructure you are using, but it's possible that you don't even have to use x-cloud-trace-context.

On using other than HttpContext for establishing a trace context: We have Google.Cloud.Diagnostics.AspNetCore[3].IExternalTraceProvider that you can use to provide (for logging) any tracing context you'd like. This will solve both your issues of not having to depend on HttpContext and being able to read some other than x-cloud-trace-context trace headers (as long as the infrastructure sends them our way of course). Usage of this is not documented (I can work on that) but you can take a look at #5214 which introduced this feature. Basically you configure the logging component of Google.Cloud.Diagnostics.AspNetCore[3] but not the tracing component, and then, with standard DI, inject whatever implementation of IExternalTraceProvider you need. In the absence of Google.Cloud.Diagnostics.AspNetCore[3]'s tracing component, we'll use that provider for obtaining trace context for log entries. There's also FR #5360 open for adding trace information to error log entries logged with the error reporting component of Google.Cloud.Diagnostics.AspNetCore[3] in case you were planning on using that.

On Activity (I think you mean System.Diagnostic.Activity): Yes, this probably is a nice way to go, but it will require lots of work now, and since we are in-between current state and the hope of OpenTelemetry integration, we probably won't be investing time on it unless there's absolutely no other alternative. I believe what I've described above is a good alternative for you.

Let me know.

[jstafford5380]

@amanda-tarafa Sorry for the delay, I missed the response email...

I see and that's a little disheartening to hear, for sure.

On IExternalTraceProvider (I'm still trying wrap my head around it). It seems that what you're saying is just to implement that interface and use it where needed. I'm trying to imagine what "where needed" would end up looking like. Say a message came in on pub/sub and it had a trace ID. When it is received, the first thing I would want to do is establish an Activity before dispatching that message to its handlers. From that point on, I would expect that any tracing or logging components would reference that Activity for the duration. If I were to implement IExternalTraceProvider to maybe look at Activity.Current.TraceStateString and use that to assemble TraceContextForLogEntry that sounds like it would ensure the log gets it, but would the trace? Would the propagating trace handler use this to ensure this is passed on to any subsequent outbound http calls?

Ideally, what I would want to see is that establishing a trace when there is no http context should be the same as when there is an http context. That is, once it's established, the rest of the pipeline, so to speak, will continue on as normal. This is of course to avoid any sort of "modal setup" that developers would be expected to obey. I'd hate to have to develop a message handler or service in code that has to check whether or not it's processing a request in an HTTP context or a non-HTTP context so that it knows whether or not it needs to insert a custom HTTP handler before making an outbound request, just to ensure a trace is continued, if that makes sense.

[amanda-tarafa]

Yes, IExternalTraceProvider is only intended to work for the Logging component of the Diagnostics libraries. I wasn't sure from your original post whether you wanted to expand traces or just add the information to log entries.

If I'm understanding you correctly, you want an Activity based layer that can be initialized from multiple sources, one of them being HTTP, one of them being Pub/Sub subscribers, maybe some others. Google.Cloud.Diagnostics.AspNetCore* is not that layer, nor is something you could simply wrap with Activity because it is, as @jskeet said, naturally tied to HttpContext.

But, I think you can use Goole.Cloud.Diagnostics.AspNetCore* as the implementation of the HTTP dependand bits, and Google.Cloud.Diagnostics.Common for implementing the rest:

• At the center is Diagnostics.Common.ContextTraceManager which holds the Common.IManagedTracer for the current context (context here used in general terms). You'd want to write your Activity layer on top of that, so that Activity writes to and reads from the current Common.IManagedTracer.
• You'd configure the tracing Diagnostics.AspNetCore* so that the middleware defined there handles HTTP context based tracing, i.e., initializes the current IManagedTracer correctly for HTTP requests. This would be currently tied to Google's header format, but I can add a simple hook here so you can use the W3C format if needed.
• You'd then write code that initializes IManagedTracer for non-HTTP context tracing. For that you can obtain a factory Func<TraceHeaderContext, IManagedTracer> from standard DI (it was registered by Diagnostics.AspNetCore*) and you'd use that factory and TraceHeaderContext.Create(string traceId, ulong? spanId, bool? shouldTrace) to obtain an IManagedTracer that you can set as the current tracer on ContextTraceManager, for Pub\Sub incoming messages or anything else.
• From then on everything should work the same.

This might seem like a little too much, but is really not. Notice that you don't even need to write the Activity based layer if you are OK with using IManagedTracer directly. And the other bit that you need to write is really the initialization of the trace context for non-HTTP based traces, which is expected being custom (with respect to what we support).

If I didn't understand right, I'd need a little more context, maybe some code snippets of what you want to achieve to try and help you better.

Let me know.

[jstafford5380]

This might seem like a little too much, but is really not.

No, not at all! I think that makes sense. I think I would need to figure out how to create get traceid, ulong, spanid from a trace traceparent though in order to create that context.

[snakefoot]

Microsoft have created System.DiagnosticSource.Activity that allows you to capture TraceId and SpanId.

See also: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md

[StasPerekrestov]

BTW, OpenTelemetry .NET reaches v1.0 https://devblogs.microsoft.com/dotnet/opentelemetry-net-reaches-v1-0/

[amanda-tarafa]

Thanks @StasPerekrestov. Yes we are already looking into next steps. An annoucement will be made once we have a plan.

[amanda-tarafa]

@jstafford5380 Given your latest comments on #5929 I was wondering if you still have the need for this feature or not, i.e. the hook to provide trace information that is not comming from Google's own trace header.

@StasPerekrestov you've also shown interest. Is my proposed approach in #5897 (comment) something that would work for you? Would you need the hook as well for obaining the trace information from other than Google's own trace header?

I'm just prioritizing the Diagnostics work and this info will help me do that. Thanks.

[jstafford5380]

Yeah, I think it's still needed. We still need the ability to re-establish a trace context that didn't necessarily come in from an http request.

…

On Tue, Apr 6, 2021 at 8:45 AM Amanda Tarafa Mas ***@***.***> wrote: @jstafford5380 <https://github.com/jstafford5380> Given your latest comments on #5929 <#5929> I was wondering if you still have the need for this feature or not, i.e. the hook to provide trace information that is not comming from Google's own trace header. @StasPerekrestov <https://github.com/StasPerekrestov> you've also shown interest. Is my proposed approach in #5897 (comment) <#5897 (comment)> something that would work for you? Would you need the hook as well for obaining the trace information from other than Google's own trace header? I'm just prioritizing the Diagnostics work and this info will help me do that. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5897 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALI4REDEUCWV7Q7WVPDPIELTHMUDDANCNFSM4W5OJ3JQ> .

[StasPerekrestov]

In my case, I use a subscription to pubnub and there is no HttpContext when I receive a message.

[amanda-tarafa]

OK, thanks for the info. I'll work on this then.

[amanda-tarafa]

I've just merged #6568 that addresses this issue fully by:

• Decoupling Diagnostics from Google's trace header so you can use other trace context formats.
• Making it easier to use the Diagnostics packages from non ASP.NET Core apps, in particular when requests are not HTTP requests.
• I've added docs and simple code snippets for both use cases.

I'll flag here when we've released, I first want to addres a couple more requests that we have open.