Suppress instrumentation in Zipkin exporter #1150

alanwest · 2020-08-24T21:04:16Z

Changes

We had previously discussed that it should be up to the exporter author to deal with instrumentation suppression (see: #1098 (comment)). This PR suppresses instrumentation from Zipkin.

For significant contributions please make sure you have completed the following items:

Design discussion issue #
Changes in public API reviewed

reyang

LGTM.

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs

alanwest · 2020-08-24T21:12:55Z

~~Cool! And I think this PR may have also highlighted a bug in the RuntimeContext stuff for the net452 test. Taking a look at this now...~~

There is a bug, but I think it is in the HttpWebRequest instrumentation. It has it's own ActivitySource and even though instrumentation is suppressed StartActivity returns non null.

Ah, found it. The ZipkinExportTests adds its own listener:

opentelemetry-dotnet/test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs

Lines 47 to 54 in f6ef67b

    
           var listener = new ActivityListener 
        
           { 
        
               ShouldListenTo = _ => true, 
        
               GetRequestedDataUsingParentId = (ref ActivityCreationOptions<string> options) => ActivityDataRequest.AllData, 
        
               GetRequestedDataUsingContext = (ref ActivityCreationOptions<ActivityContext> options) => ActivityDataRequest.AllData, 
        
           }; 
        
           ActivitySource.AddActivityListener(listener);

Ok, what I learned from this is that I think our Sdk.SuppressInstrumentation checks in ActivityListener.GetRequestedDataUsingContext is insufficient to suppress instrumentation from ActivitySource based activities. That is, if any listener responds with ActivityDataRequest.AllDataAndRecorded then ActivityStarted and ActivityStopped will fire on the listener created by the SDK. So, I think this means we should add checks in these callbacks too.

Is this the expected behavior from the .NET side? That is, if:

ActivityListener A says ActivityDataRequest.None, and
ActivityListener B says ActivityDataRequest.AllData, then
should A's ActivityStarted and ActivityStopped events get fired?

@cijothomas @reyang @CodeBlanch

…/ActivityStopped

codecov · 2020-08-24T23:15:53Z

Codecov Report

Merging #1150 into master will increase coverage by 0.31%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1150      +/-   ##
==========================================
+ Coverage   77.10%   77.41%   +0.31%     
==========================================
  Files         219      219              
  Lines        6259     6253       -6     
==========================================
+ Hits         4826     4841      +15     
+ Misses       1433     1412      -21

Impacted Files	Coverage Δ
...mentation.Http/HttpClientInstrumentationOptions.cs	`100.00% <ø> (ø)`
...rc/OpenTelemetry.Exporter.Zipkin/ZipkinExporter.cs	`87.01% <100.00%> (+0.17%)`	⬆️
src/OpenTelemetry/Trace/TracerProviderSdk.cs	`94.69% <100.00%> (ø)`
...plementation/ZipkinActivityConversionExtensions.cs	`91.20% <0.00%> (+1.09%)`	⬆️
...ter.ZPages/Implementation/ZPagesActivityTracker.cs	`100.00% <0.00%> (+2.85%)`	⬆️
...nTelemetry/Internal/OpenTelemetrySdkEventSource.cs	`80.95% <0.00%> (+3.17%)`	⬆️
...penTelemetry/Trace/BatchExportActivityProcessor.cs	`57.33% <0.00%> (+22.66%)`	⬆️

cijothomas · 2020-08-24T23:47:09Z

src/OpenTelemetry/Trace/TracerProviderSdk.cs

@@ -68,7 +68,7 @@ static TracerProviderSdk()
                // Callback when Activity is started.
                ActivityStarted = (activity) =>
                {
-                    if (activity.IsAllDataRequested)
+                    if (!Sdk.SuppressInstrumentation && activity.IsAllDataRequested)


we shouldn't need this as we already prevented the activity from being created in the 1st place right?

See the info I left about this above ⬆️. Basically, if any listener says 👍, then all listeners ActivityStarted/ActivityStopped gets fired. I posed the question of whether this is the expected behavior from .NET.

Opened dotnet/runtime#41306 to validate whether this is expected or not.

Great find @alanwest. I seem to remember seeing some discussion about that behavior somewhere and that is the correct behavior but let's see what they say.

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs

reyang · 2020-08-25T00:44:39Z

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs

+            var zipkinExporter = new ZipkinExporter(exporterOptions);
+            var exportActivityProcessor = new BatchExportActivityProcessor(zipkinExporter, scheduledDelayMilliseconds: 1);
+
+            var openTelemetrySdk = Sdk.CreateTracerProviderBuilder()


@cijothomas I guess we want to use traceProvider everywhere?

yes. I'll doing mass cleanup for other places.

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs

…:alanwest/opentelemetry-dotnet into alanwest/zipkin-suppress-instrumentation

cijothomas · 2020-08-25T05:09:09Z

@alanwest sorry this got delayed and has conflict now :) PLease resolve and we are good to go.

…uppress-instrumentation

CodeBlanch · 2020-08-26T04:14:23Z

src/OpenTelemetry.Instrumentation.Http/HttpClientInstrumentationOptions.cs

@@ -44,32 +44,10 @@ public class HttpClientInstrumentationOptions
        /// </summary>
        public Func<HttpRequestMessage, bool> FilterFunc { get; set; }

-        internal static bool IsInternalUrl(Uri requestUri)


It was a long road but we finally got this removed! 🎊

CodeBlanch · 2020-08-26T04:20:04Z

src/OpenTelemetry.Instrumentation.Http/HttpClientInstrumentationOptions.cs

        internal bool EventFilter(string activityName, object arg1)
        {
            if (TryParseHttpRequestMessage(activityName, arg1, out HttpRequestMessage requestMessage))
            {
-                Uri requestUri;


@cijothomas @alanwest @eddynaka We could probably improve the perf on this now by changing the order of things.

internal bool EventFilter(string activityName, object arg1) { return (this.FilterFunc == null || !TryParseHttpRequestMessage(activityName, arg1, out HttpRequestMessage requestMessage)) ? true : this.FilterFunc(requestMessage); }

Maybe also add the inline hint on this & TryParseHttpRequestMessage?

Suppress instrumentation in Zipkin exporter

aadfedb

alanwest requested a review from a team August 24, 2020 21:04

reyang approved these changes Aug 24, 2020

View reviewed changes

alanwest commented Aug 24, 2020

View reviewed changes

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs Outdated Show resolved Hide resolved

alanwest added 2 commits August 24, 2020 15:49

Check Sdk.SuppressInstrumentation on ActivityListener.ActivityStarted…

cafd1b1

…/ActivityStopped

Remove raw post in test to Zipkin endpoint URL

2032ab9

cijothomas reviewed Aug 24, 2020

View reviewed changes

Merge branch 'master' into alanwest/zipkin-suppress-instrumentation

ae962c8

reyang reviewed Aug 25, 2020

View reviewed changes

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs Outdated Show resolved Hide resolved

reyang reviewed Aug 25, 2020

View reviewed changes

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs Outdated Show resolved Hide resolved

reyang reviewed Aug 25, 2020

View reviewed changes

reyang self-requested a review August 25, 2020 00:45

cijothomas reviewed Aug 25, 2020

View reviewed changes

test/OpenTelemetry.Exporter.Zipkin.Tests/ZipkinExporterTests.cs Outdated Show resolved Hide resolved

alanwest and others added 4 commits August 24, 2020 18:14

Use ForceFlush on exporter

2f0db5e

Merge branch 'master' into alanwest/zipkin-suppress-instrumentation

281d5bb

Call ForceFlush twice

c461b0a

Merge branch 'alanwest/zipkin-suppress-instrumentation' of github.com…

093f8d7

…:alanwest/opentelemetry-dotnet into alanwest/zipkin-suppress-instrumentation

alanwest mentioned this pull request Aug 25, 2020

Removing Application Insights legacy code #1160

Merged

2 tasks

cijothomas closed this Aug 25, 2020

cijothomas reopened this Aug 25, 2020

alanwest added 2 commits August 25, 2020 09:13

Merge remote-tracking branch 'upstream/master' into alanwest/zipkin-s…

6b1e15c

…uppress-instrumentation

Remove IsInternalUrl

bc067ee

cijothomas approved these changes Aug 25, 2020

View reviewed changes

cijothomas merged commit c21bd58 into open-telemetry:master Aug 25, 2020

alanwest deleted the alanwest/zipkin-suppress-instrumentation branch August 25, 2020 17:01

CodeBlanch reviewed Aug 26, 2020

View reviewed changes

alanwest mentioned this pull request Aug 26, 2020

Optimize HttpClientInstrumentationOptions.EventFilter #1170

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suppress instrumentation in Zipkin exporter #1150

Suppress instrumentation in Zipkin exporter #1150

alanwest commented Aug 24, 2020 •

edited

Loading

reyang left a comment

alanwest commented Aug 24, 2020 •

edited

Loading

codecov bot commented Aug 24, 2020 •

edited

Loading

cijothomas Aug 24, 2020

alanwest Aug 25, 2020 •

edited

Loading

alanwest Aug 25, 2020

CodeBlanch Aug 25, 2020

reyang Aug 25, 2020

cijothomas Aug 25, 2020

cijothomas commented Aug 25, 2020

CodeBlanch Aug 26, 2020

CodeBlanch Aug 26, 2020

Suppress instrumentation in Zipkin exporter #1150

Suppress instrumentation in Zipkin exporter #1150

Conversation

alanwest commented Aug 24, 2020 • edited Loading

Changes

reyang left a comment

Choose a reason for hiding this comment

alanwest commented Aug 24, 2020 • edited Loading

codecov bot commented Aug 24, 2020 • edited Loading

Codecov Report

cijothomas Aug 24, 2020

Choose a reason for hiding this comment

alanwest Aug 25, 2020 • edited Loading

Choose a reason for hiding this comment

alanwest Aug 25, 2020

Choose a reason for hiding this comment

CodeBlanch Aug 25, 2020

Choose a reason for hiding this comment

reyang Aug 25, 2020

Choose a reason for hiding this comment

cijothomas Aug 25, 2020

Choose a reason for hiding this comment

cijothomas commented Aug 25, 2020

CodeBlanch Aug 26, 2020

Choose a reason for hiding this comment

CodeBlanch Aug 26, 2020

Choose a reason for hiding this comment

alanwest commented Aug 24, 2020 •

edited

Loading

alanwest commented Aug 24, 2020 •

edited

Loading

codecov bot commented Aug 24, 2020 •

edited

Loading

alanwest Aug 25, 2020 •

edited

Loading