Sampling when span has links

[anuraaga]

Currently the ParentBased sampler only defines behavior for remote parents. Should it also define behavior that looks at Links? Or even if it's not appropriate for the standard ParentBased sampler, does anyone have any recommendations for sampling in the context of links? For example, one approach could be

• If all links have sampled = false, don't sample
• If any link has sampled = true, sample
• If no links, make a sampling decision

I'm wondering if anything like this should be added to the spec, or if anyone has any pointers for this.

[pyohannes]

My team was facing this problem recently. We went with using a separate LinkBasedSampler, which samples if any link is sampled (but this could be made configurable).

So we ended up with a sampler chain: ParentBased -> LinkBased -> TraceIdRatio. We found this to be more flexible, as compared to adding the logic to the ParentBased sampler.

On the downside, this makes sampling volume harder to predict. And it doesn't consider links to non-root spans. A tail-based sampling approach would be needed to cover that.

[anuraaga]

Thanks for sharing @pyohannes! Just to confirm, does the LinkBasedSampler drop spans if it has links but they were all unsampled? If not, any reasoning behind falling back to the TraceIdRatio? I think whether there's such a fallback or not is case-by-case but curious if you have any anecdote.

Coincidentally I happened to implement a package-private OrElseSampler which is probably how the arrows in your diagram are implemented, maybe this could be added to the spec (and relates to #1844)

https://github.com/open-telemetry/opentelemetry-java-contrib/pull/68/files#diff-9485e048e347074485126371a20d3a43ea3e885d74d5debb7674d0a89b22a7cd

[pyohannes]

Thanks for sharing @pyohannes! Just to confirm, does the LinkBasedSampler drop spans if it has links but they were all unsampled? If not, any reasoning behind falling back to the TraceIdRatio? I think whether there's such a fallback or not is case-by-case but curious if you have any anecdote.

We're falling back to the TraceIdRatio sampler for root spans without any links. This works well with our scenarios, but definitely needs much more thought for making this a general solution. Probably the exact behavior needs to be configurable.

Coincidentally I happened to implement a package-private OrElseSampler which is probably how the arrows in your diagram are implemented, maybe this could be added to the spec

We didn't need any OrElseSampler. LinkBased and ParentBased have this logic baked in.

builder.SetSampler(new ParentBasedSampler(new LinkBasedSampler(new TraceIdRatioBasedSampler(ratio))));

We followed the logic of the ParentBased sampler for the LinkBased sampler.