-
Notifications
You must be signed in to change notification settings - Fork 894
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove warnings for trace.UpdateName #506
remove warnings for trace.UpdateName #506
Conversation
Fixes open-telemetry#468 The section discouraging trace.UpdateName clear in terms of the reasoning, but practically it's common for consumers to not read the full specification or documentation. As such, one should expect that UpdateName use will not be uncommon, and those who process spans cannot rely on the span name being immutable. Removing the section discouraging it, and adding a section to the samplers themselves to ensure that samplers are aware of mutable span names.
Now that we have normalized its usage, I would propose we replace |
Yes! Will follow up this PR with a subsequent one for that. |
I don't quite follow the reasoning for removing the warning. Samplers have to deal with changing span name anyway, but it may lead to more expensive implementation than in cases where the name is provided upfront. For example, in Jaeger it's possible to configure the sampler to expect that the name won't change after creation, leading to a more efficient operation. And in many cases instrumentation does have options for not relying on UpdateName, e.g. by deferring span creation. So I feel that the paragraph discouraging the use of UpdateName was useful. |
I want to say that this issue is not very important to me, but looks like a major wrinkle from the user's perspective. I don't like the way that name has been made out to be a special case. It's true that some applications cannot know their name until after the span starts, but it's also true that some span attributes are not known until after the span starts. We caution in the specification that span attributes set after start may not be used in the sampling decision, but we carve out a special case for span names. If a span attribute or the span name can change multiple times before it ends, and you wish to continually update the sampling decision, then what you're describing is tail sampling. The cost of tail sampling is too high, you say? That's why we have head sampling, it's simple and cheap. The complaint is that head sampling is what you want except for the irritating fact that sometimes a name is not known at the head of a span. If head sampling isn't working, either use tail sampling or use a more sophisticated (*) For example, a |
Another alternative is to allow not setting the name at start when it is known that the final name is unknown. |
Yes. I would be much more comfortable if we allowed the name to be set exactly once for the Sampler--either when the span is started or at one later point in time. Then, just like |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change LGTM for the reasons @jmacd points out, and in general I think it's better for the spec to describe how implementations ought to be written than how users ought to use them. That is, if we provide an UpdateName
API method the onus should be on the author of the sampler to explain that changes to the name won't affect the sampling decision, not on the user to know not to use that particular method.
We caution in the specification that span attributes set after start may not be used in the sampling decision, but we carve out a special case for span names.
We did something similar for links after OTEP 0006. From the user's perspective, it is surprising that the span name and attributes can be changed after creation, but not its links.
specification/sdk-tracing.md
Outdated
#### Sampler Design Considerations | ||
|
||
Sampling algorithms should consider the fact that many attributes of the span | ||
are mutable after it's initial creation, including span attributes and the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are mutable after it's initial creation, including span attributes and the | |
are mutable after its initial creation, including span attributes and the |
It's true in some cases it is possible to ensure that the span is included in the name, by refactoring the code. But I think that could also be said about other things like span attributes as well, which, to @jmacd's point, don't have any special casing around them.
I think that's the issue, and it goes beyond pushing complexity to the user. Practically speaking, most people will not read the full specification, and will probably use their language's API documentation or some IDE autocomplete to discover UpdateName. So many people will not even be aware of the fact that UpdateName is a discouraged API. I worry that the existing terminology will make the authors who write sampling code to believe they can rely on reduced usage of UpdateName and handle that case differently, or not at all. That then causes the situation where end users see unexpected sampling logic because of their use of UpdateName, file an issue with the sampler author, and be told that they should not be using that API. It's a realm of complexity that we can avoid if we push the caveat that the value is mutable into the sampling side.
Agreed. I think that's more impetus to put these caveats on the sampler side.
I think this is hard behavior to enforce in practice, but conceptually that make sense. |
What if we rename it to |
For what it's worth, it's not clear to me that "overwriteName" is any more dangerous that "updateName" - might just be adjective classifications I'm not really familiar with. |
I spoke with @bogdandrutu, who asked for some clarification on why we couldn't remove this API from the specification, which would simplify things if possible. One major use case is when http requests are started before a canonical span name is provided. @SergeyKanzhelev has highlighted this for C#: #468 (comment) The general idea applies to any language: A common span name to use is the "route" name in the web framework, such as a templatized route
The alternative to providing updateName is to store the start time and other data in some object, and use that in the creation of the span once metadata is read. To enable that, that would require a pattern where such span data is stored is some temporary object (probably in the context) before span creation. IMO this is a much more complex pattern to facilitate, and would then require a second convention on where to store this span data pre-span creation. Next StepsAs I understand it, the main concern is the fact that the Sampler API is designed such that it is not possible to re-evaluate the early sampling decision based on changes to the span (or it is expensive to do so). I think this can come in as a separate PR, but how about adding APIs to the Sampler in the spec to handle changes to the span as they come in? That would handle a currently unhandled edge case of span attributes changing throughout the lifetime of the span too. This is especially useful on things like response code, which is unknown until near the span end itself. |
@SergeyKanzhelev question for you: before this PR, the strategy to store the span start time and overwrite start time was called out in the spec:
It sounds like ASP.net has explicitly chosen to use UpdateName over this strategy. What was the rationale? If we don't have official opentelemetry packages following the recommendations in the spec, I feel like we need to change the spec, or the implementation. |
FWIW, Python's flask instrumentation also uses the store timestamp + start later approach and has to go to quite some lengths (boilerplate-wise) for that. EDIT: But I view that as a Quality-of-Implementation thing. You can take the shortcut of calling UpdateName to get more maintainable instrumentation code, or you can write some more complicated, longer instrumentation to be able to provide better information to the sampler. |
This opens a can of worms, like what to do if child spans with that span's ID as parent have already been created (even exported) and now that span is sampled away? The parent-child chain in the trace would be broken. EDIT: #307 (still open) discusses this. |
Please, can we keep the scope of our discussion to the scope of this PR? I support the opinion that we cannot remove I don't think that span name differs in any way from any other span attribute from Sampler perspective. So if sampler deals somehow with attribute changes, it has to deal with span name change as well. This comment from @c24t summarises it very nicely:
For these reasons I think we have to merge this PR. As long as we have @toumorokoshi Can you please rebase the PR? @open-telemetry/specs-approvers @open-telemetry/specs-trace-approvers If you agree with this PR and my argument above, please approve this PR. |
@@ -120,6 +120,12 @@ These are the default samplers implemented in the OpenTelemetry SDK: | |||
should be configuration to change this to "root spans only", or "all spans". | |||
* Description MUST be `ProbabilitySampler{0.000100}`. | |||
|
|||
#### Sampler Design Considerations | |||
|
|||
Sampling algorithms should consider the fact that many attributes of the span |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does "consider" mean here? I don't think samplers can sensibly do anything about this. I think we should rather write for SetAttribute and UpdateName (I think we have something like that in the span creation API spec already):
Note that samplers can only consider information already set during span creation. Any changes done later cannot be considered for sampling.
Closing on behalf of #754 |
Fixes #468
The section discouraging trace.UpdateName clear in terms of the reasoning,
but practically it's common for consumers to not read the full specification
or documentation. As such, one should expect that UpdateName use will not
be uncommon, and those who process spans cannot rely on the span name being
immutable.
Removing the section discouraging it, and adding a section to the samplers
themselves to ensure that samplers are aware of mutable span names.