Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v3.5] Change the default sampling rate from 100% to 0% #14310

Closed
dashpole opened this issue Aug 4, 2022 · 2 comments
Closed

[v3.5] Change the default sampling rate from 100% to 0% #14310

dashpole opened this issue Aug 4, 2022 · 2 comments

Comments

@dashpole
Copy link

dashpole commented Aug 4, 2022

Part of #12460.

Etcd, especially in a kubernetes cluster, handles very high QPS. In the initial version (#12919) of OpenTelemetry tracing, it did not set a sampler, which causes the sampling rate to default to sampling every request. This makes tracing virtually unusable on etcd 3.5.

When a sampler was added in 3.6, it changed to having it default to ParentBased(0), which is the preferred default.

I'd like to propose backporting this line, with ParentBased(NeverSample()) to release-3.5 to address this.

cc @logicalhan @damemi @CatherineF-dev @dgrisonnet @serathius @sallyom

@damemi
Copy link

damemi commented Aug 5, 2022

Willing to work on this if no one else is already

damemi added a commit to damemi/etcd that referenced this issue Aug 5, 2022
This changes the default parent-based trace sampling rate from
100% to 0%. Due to the high QPS etcd can handle, having 100% trace
sampling leads to very high resource usage. Defaulting to 0% means
that only already-sampled traces will be sampled in etcd.

Fixes etcd-io#14310
damemi added a commit to damemi/etcd that referenced this issue Aug 5, 2022
This changes the default parent-based trace sampling rate from
100% to 0%. Due to the high QPS etcd can handle, having 100% trace
sampling leads to very high resource usage. Defaulting to 0% means
that only already-sampled traces will be sampled in etcd.

Fixes etcd-io#14310

Signed-off-by: Mike Dame <[email protected]>
@serathius
Copy link
Member

serathius commented Aug 7, 2022

fixed by #14318

openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/etcd that referenced this issue Oct 7, 2022
This changes the default parent-based trace sampling rate from
100% to 0%. Due to the high QPS etcd can handle, having 100% trace
sampling leads to very high resource usage. Defaulting to 0% means
that only already-sampled traces will be sampled in etcd.

Fixes etcd-io#14310

Signed-off-by: Mike Dame <[email protected]>
tjungblu pushed a commit to tjungblu/etcd that referenced this issue Jul 26, 2023
This changes the default parent-based trace sampling rate from
100% to 0%. Due to the high QPS etcd can handle, having 100% trace
sampling leads to very high resource usage. Defaulting to 0% means
that only already-sampled traces will be sampled in etcd.

Fixes etcd-io#14310

Signed-off-by: Mike Dame <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants