Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: avoid lease transfers to restarted nodes that could become overloaded #96521

Closed
2 tasks done
kvoli opened this issue Feb 3, 2023 · 3 comments
Closed
2 tasks done

kvserver: avoid lease transfers to restarted nodes that could become overloaded #96521

kvoli opened this issue Feb 3, 2023 · 3 comments
Assignees
@kvoli
Labels
A-kv-distribution Relating to rebalancing and leasing. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-kv KV Team
Milestone
23.1

Comments

@kvoli
Copy link
Collaborator

kvoli commented Feb 3, 2023 •
edited
Loading

We have seen that a node restarting in the cluster can have very negative effects on request latency.

A reproduction test was added in #95161

Allowing Lease Transfers
Extremely High P99 SQL Latency
image
image

Blocking Lease Transfers
No impact on P99 SQL Latency
image
image

This issue tracks the improvements to crdb in 23.1 to prevent a restarted node impacting end latency. The improvements are narrowly scoped but have been proven to have a significant effect in keeping request latency normal.

Jira issue: CRDB-24164
@kvoli kvoli added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-distribution Relating to rebalancing and leasing. labels Feb 3, 2023
@kvoli kvoli added this to the 23.1 milestone Feb 3, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Feb 3, 2023
@kvoli kvoli changed the title kvserver: node restarting should not impact request latency kvserver: avoid lease transfers to restarted nodes that could become overloaded Feb 3, 2023
@kvoli
Copy link
Collaborator Author

kvoli commented Feb 3, 2023 •
edited
Loading

xref #95159

@kvoli kvoli self-assigned this Feb 6, 2023
This was referenced Feb 16, 2023
Open
Closed
Open
@blathers-crl
Copy link

blathers-crl bot commented Mar 2, 2023

Hi @kvoli, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@kvoli kvoli added the branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 label Mar 6, 2023
@kvoli
Copy link
Collaborator Author

kvoli commented Mar 8, 2023

Closed by completing #96508.

@kvoli kvoli closed this as completed Mar 8, 2023
@kvoli kvoli mentioned this issue Mar 9, 2023
Closed
kvoli added a commit to kvoli/cockroach that referenced this issue Mar 9, 2023
@kvoli
roachtest: unskip kv/restart/nodes=12
b3d3e64 
Previously, the `kv/restart/nodes=12` roachtest was unable to pass and
was skipped. Following the changes introduced for cockroachdb#96521, the test now
passes.

This commit enables the `kv/restart/nodes=12` roachtest as a weekly
test.

resolves: cockroachdb#98296

Release note: None
@kvoli kvoli mentioned this issue Mar 9, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 9, 2023
@kvoli
Merge #98271
deb3f94 
98271: roachtest: unskip kv/restart/nodes=12 r=andrewbaptist a=kvoli

Previously, the `kv/restart/nodes=12` roachtest was unable to pass and
was skipped. Following the changes introduced for #96521, the test now
passes.

This commit enables the `kv/restart/nodes=12` roachtest as a weekly
test.

resolves: #98296

Release note: None

Co-authored-by: Austen McClernon <[email protected]>
@irfansharif irfansharif mentioned this issue Mar 15, 2023
Open
@kvoli kvoli mentioned this issue Dec 5, 2023
Closed
@kvoli kvoli mentioned this issue Oct 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvoli kvoli

Labels
A-kv-distribution Relating to rebalancing and leasing. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-kv KV Team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
1 participant
@kvoli