-
Notifications
You must be signed in to change notification settings - Fork 773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SuppressInstrumentation from ActivityListener and DiagnosticSourceListener #1079
SuppressInstrumentation from ActivityListener and DiagnosticSourceListener #1079
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1079 +/- ##
==========================================
+ Coverage 74.63% 74.76% +0.12%
==========================================
Files 223 223
Lines 6360 6365 +5
==========================================
+ Hits 4747 4759 +12
+ Misses 1613 1606 -7
|
@@ -39,6 +39,11 @@ public void OnError(Exception error) | |||
|
|||
public void OnNext(KeyValuePair<string, object> value) | |||
{ | |||
if (Sdk.SuppressInstrumentation) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably a good optimization to have this, but I don't think it's strictly required. For "legacy" Activity flows we pass them through an ActivitySource for sampling. So the logic below should also catch these?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@CodeBlanch Hmm 🤔, maybe I need you to point me to some code... because without this check here, this test I wrote would fail:
So, it would seem that for legacy activities, we'll need a check somewhere in addition to the check I have in the ActivityListener for the new activities.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, my bad. That was a test and you passed, great job!
It looks like ActivitySourceAdapter has its own sampler, so we'll need to do similar in here:
opentelemetry-dotnet/src/OpenTelemetry/Trace/ActivitySourceAdapter.cs
Lines 85 to 131 in 44b55d4
private void RunGetRequestedData(Activity activity) | |
{ | |
ActivityContext parentContext; | |
if (string.IsNullOrEmpty(activity.ParentId)) | |
{ | |
parentContext = default; | |
} | |
else if (activity.Parent != null) | |
{ | |
parentContext = activity.Parent.Context; | |
} | |
else | |
{ | |
parentContext = new ActivityContext( | |
activity.TraceId, | |
activity.ParentSpanId, | |
activity.ActivityTraceFlags, | |
activity.TraceStateString, | |
isRemote: true); | |
} | |
var samplingParameters = new SamplingParameters( | |
parentContext, | |
activity.TraceId, | |
activity.DisplayName, | |
activity.Kind, | |
activity.TagObjects, | |
activity.Links); | |
var samplingResult = this.sampler.ShouldSample(samplingParameters); | |
switch (samplingResult.Decision) | |
{ | |
case SamplingDecision.NotRecord: | |
activity.IsAllDataRequested = false; | |
break; | |
case SamplingDecision.Record: | |
activity.IsAllDataRequested = true; | |
break; | |
case SamplingDecision.RecordAndSampled: | |
activity.IsAllDataRequested = true; | |
activity.ActivityTraceFlags |= ActivityTraceFlags.Recorded; | |
break; | |
} | |
} | |
} | |
} |
Or maybe we can refactor so more is shared?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was a test and you passed, great job!
Ha! Phew!
It looks like ActivitySourceAdapter has its own sampler, so we'll need to do similar in here
I don't think so because this is actually downstream of the check I have in place.
So, DiagnosticSourceListener.OnNext
--- invokes ---> ListenerHandler.On[Start|End|Exception|Custom]
It is our ListenerHandler
s that hold an instance of the ActivitySourceAdapter
which is then the thing that simulates Start/Stop/etc and deals with sampling like the standard ActivitySource/ActivityListener.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you saying it wouldn't work, or just that this spot is sooner? If the later, I suppose that is compelling enough a reason to keep it here and I'm good with it 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would work for preventing this.activityProcessor.OnStart(activity);
, but the reason I put the check where I did in DiagnosticSourceListener.OnNext
is that the check there prevents all instrumentation callbacks defined on the ListenerHandler
from being invoked. For example, if some library had:
myDiagnosticListener.Write("MyCustomEvent", payload);
If we had a ListenerHander
wired up to this library then this Write
would result in invoking the OnCustom
callback. I figured it would be good to suppress all aspects of the library's instrumentation.
Though I guess in most (if not all) of these callbacks we check activity.IsAllDataRequested
, so they'd result in noops anyways...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense good call!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR applies Sdk.SuppressInstrumentation originally introduced in #960.
Changes
Sdk.SuppressInstrumentation
inActivityListener
covering the newActivitySource
sourced activities.Sdk.SuppressInstrumentation
inDiagnosticSourceListener
covering the legacy activities.Some performance analysis was done on this implementation in #1049
Alternative to this PR
Given that this PR suppresses instrumentation in a general way, any perf hit from checkingSdk.SuppressInstrumentation
is incurred for everyActivity
. Alternative would be to narrow what we suppress as suggested here #977 (comment). I implemented this alternative in #1098.Based on discussion here: #1098 (comment), I think we're going to go with this PR and close #1098.
Follow up after this PR
SuppressInstrumentation = true
in Zipkin/Jaeger exporters.IsInternalUrl
from HTTP instrumentation