[FEATURE REQ] Support e2e tracing for CloudEvents in Microsoft.Azure.WebJobs.Extensions.EventGrid

[lmolkova]

Library name

Microsoft.Azure.WebJobs.Extensions.EventGrid

Please describe the feature.

EventGrid client SDKs populate distributed tracing extension with trace context. To leverage it and enable proper e2e tracing in Functions and WebJobs, the extension should read this context and continue the trace

Below is the code users need to write now (specific to Application Insights, but Microsoft.Azure.WebJobs.Extensions.EventGrid can use System.Diagnostics.Activity only and Functions already do the trick with Application Insights).

For IAsyncCollector<CloudEvent> overload, it gets tricky and we'd have to use ActivitySource to properly populate links.
Users would likely want to be able to extract context for individual events, it's tracked here Azure/azure-sdk#3319

private readonly TelemetryClient telemetryClient;
public Function1(TelemetryClient telemetryClient)
{
    this.telemetryClient = telemetryClient;
}

[FunctionName("Function1")]
public void Run([EventGridTrigger]CloudEvent evnt, ILogger log)
{
    using (var processingOperation = telemetryClient.StartOperation<RequestTelemetry>(ExtractActivity(evnt)))
    {
        // do something
        log.LogInformation("Processing  CloudEvent... {event}", evnt.Data.ToString());
    }
}

private static Activity ExtractActivity(CloudEvent evnt)
{
    // read distributed tracing extension
    // https://github.com/cloudevents/spec/blob/v1.0.1/extensions/distributed-tracing.md
    string traceparentStr = null;
    string tracestateStr = null;
    if (evnt.ExtensionAttributes.TryGetValue("traceparent", out var traceparent)
        && traceparent is string)
    {
        traceparentStr = (string) traceparent;
        if (evnt.ExtensionAttributes.TryGetValue("tracestate", out var tracestate)
            && tracestate is string)
        {
            tracestateStr = (string)tracestate;
        }

    }

    return ExtractActivity("CloudEvents Process " + evnt.Type, traceparentStr, tracestateStr);
}

private static Activity ExtractActivity(string activityName, string traceparent, string tracestate)
{
    var activity = new Activity(activityName);

    if (!string.IsNullOrEmpty(traceparent))
    {
        // parse traceparent according to https://www.w3.org/TR/trace-context/
        if (traceparent.StartsWith("00-") && traceparent.Length == 55)
        {
            // preserve function execution as parent and add link context from the event
            // assuming we do this inside the extension code, we should just make this context parent 
           // or use `ActivitySource` with proper links support if possible
            activity.AddTag("_MS.links",
                $"[{{\"operation_Id\":\"{traceparent.Substring(3, 32)}\",\"id\":\"{traceparent.Substring(36, 16)}\"}}]");
        }

        activity.TraceStateString = tracestate;
    }

    return activity;
}

[JoshLove-msft]

This was added in #25944