release-22.1: server: retry ambiguous kv/sql operations on node startup #102245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In TestClusters with more than 4 nodes, wait for full replication doesn't take updated config into account. This commit adds new check that zone config was propagated prior to making replication state checks. Epic: none Release note: None
|
This change is |
aliher1911
changed the title
aliher1911
force-pushed
the
backport22.1-97710-100583
branch
10 times, most recently
from
97bdda1 to
5398c94
Compare
Previously, nodes could fail on startup if node itself was needed to restore quorum on ranges. This problem was caused by circuit breaker failing fast on live node when talking back to restarting node instead of waiting for raft operation to run again (and succeed). This PR adds explicit retries and also adds assertions enabled in race build that would fire if startup reaches dist sender or internal sql executor without wrapping it in startup retry helper. Release note: None
We need to have the `RunIdempotentWithRetry` calls abort if the surrounding server shuts down prematurely. This happens most frequently in tests that fail for another reason; and also in multitenancy tests with multiple tenant server side-by-side. Release note: None
aliher1911
force-pushed
the
backport22.1-97710-100583
branch
from
5398c94 to
3a217ee
Compare
tbg
reviewed
Apr 27, 2023
tbg
approved these changes
Apr 27, 2023
| @@ -479,7 +480,7 @@ func ExpectedDescriptorIDs( | |||
| defaultZoneConfig *zonepb.ZoneConfig, | |||
| defaultSystemZoneConfig *zonepb.ZoneConfig, | |||
| ) (descpb.IDs, error) { | |||
| completedMigrations, err := getCompletedMigrations(ctx, db, codec) | |||
| completedMigrations, err := getCompletedMigrations(ctx, make(chan struct{}), db, codec) | |||
There was a problem hiding this comment.
Can't passing the struct here lead to stalls during quiesce?
Ah I see the description says this is test only. I also no longer see this method on master. Ok, let's leave as is.
There was a problem hiding this comment.
Go land even shows me this method as grey and I don't see any use of it, so it is likely a leftover but I decided not to clutter the PR with unrelated change.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
server,util: avoid retry forever during startup upon premature shutdown
We need to have the
RunIdempotentWithRetrycalls abort if thesurrounding server shuts down prematurely. This happens most
frequently in tests that fail for another reason; and also
in multitenancy tests with multiple tenant server side-by-side.
Release note: None
server: retry ambiguous kv/sql operations on node startup
Previously, nodes could fail on startup if node itself was needed to
restore quorum on ranges. This problem was caused by circuit breaker
failing fast on live node when talking back to restarting node instead
of waiting for raft operation to run again (and succeed).
This PR adds explicit retries and also adds assertions enabled in race
build that would fire if startup reaches dist sender or internal sql
executor without wrapping it in startup retry helper.
Release note: None
testutil: zone config propagation wait for test cluster
In TestClusters with more than 4 nodes, wait for full replication
doesn't take updated config into account.
This commit adds new check that zone config was propagated prior to
making replication state checks.
Epic: none
Release note: None
Backport:
/cc @cockroachdb/release
Release justification: This backport addresses production issue with nodes failing start.