release-22.2: server: retry ambiguous kv/sql operations on node startup #100463
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In TestClusters with more than 4 nodes, wait for full replication doesn't take updated config into account. This commit adds new check that zone config was propagated prior to making replication state checks. Epic: none Release note: None
|
This change is |
aliher1911
changed the title
erikgrinaker
changed the title
aliher1911
force-pushed
the
backport22.2-97710
branch
8 times, most recently
from
4c7a842 to
fcbf4b7
Compare
Previously, nodes could fail on startup if node itself was needed to restore quorum on ranges. This problem was caused by circuit breaker failing fast on live node when talking back to restarting node instead of waiting for raft operation to run again (and succeed). This PR adds explicit retries and also adds assertions enabled in race build that would fire if startup reaches dist sender or internal sql executor without wrapping it in startup retry helper. Release note: None
We need to have the `RunIdempotentWithRetry` calls abort if the surrounding server shuts down prematurely. This happens most frequently in tests that fail for another reason; and also in multitenancy tests with multiple tenant server side-by-side. Release note: None
aliher1911
force-pushed
the
backport22.2-97710
branch
from
fcbf4b7 to
89855c8
Compare
|
TestTransientClusterSimulateLatencies is unrelated to backport and are flaking on the branch with the same error previously. |
|
@tbg this is a backport PR |
tbg
approved these changes
Apr 27, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In TestClusters with more than 4 nodes, wait for full replication
doesn't take updated config into account.
This commit adds new check that zone config was propagated prior to
making replication state checks.
Epic: none
Release note: None
Backport 2/2 commits from #97710.
/cc @cockroachdb/release
Previously, nodes could fail on startup if node itself was needed to restore quorum on ranges. This problem was caused by circuit breaker failing fast on live node when talking back to restarting node instead of waiting for raft operation to run again (and succeed).
This PR adds explicit retries and also adds assertions enabled in race build that would fire if startup reaches dist sender or internal sql executor without wrapping it in startup retry helper.
Release note: None
Fixes #74714
Includes fix #100583
Release justification: fixes a stability issue during node restarts