roachtest: (triaged) restore/nodeShutdown/worker failed [from transient admin split error] #84635
Assignees
Labels
branch-release-22.1
Used to mark GA and release blockers, technical advisories, and bugs for 22.1
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
stability-period
Intended to be worked on during a stability period. Use with the Milestone field to specify version.
T-disaster-recovery
Milestone
Comments
cockroach-teamcity
added
branch-release-22.1
|
The restore job failed on node 2. I see the following error in the jobs table. This smells like KV, but i will look a bit more at the logs in the morning. In node 2's logs, I see: I'm removing the release blocker because this error seems to be a retryable error: |
msbutler
removed
the
release-blocker
|
Bulk should certainly have retried the RESTORE due to this retryable error, which is easily checked via I wonder if it's worth wrapping every bulk kv request with a function that handles these temperamental kv request errors. Will add a discussion item for next week. |
msbutler
added a commit
to msbutler/cockroach
that referenced
this issue
Jul 29, 2022
Informs cockroachdb#84635,cockroachdb#84162 Release note: none
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Aug 1, 2022
…shot Informs cockroachdb#84635 Informs cockroachdb#84162 This commit considers the `LeaseTransferRejectedBecauseTargetMayNeedSnapshot` status to be a form of a retriable replication change error. It then hooks `Replica.executeAdminCommandWithDescriptor` up to consult this status in its retry loop. This avoids spurious errors when a split gets blocked behind a lateral replica move like we see in the following situation: 1. issue AdminSplit 2. range in joint config, first needs to leave (maybeLeaveAtomicChangeReplicas) 3. to leave, needs to transfer lease from voter_outgoing to voter_incoming 4. can’t transfer lease because doing so is deemed to be potentially unsafe Release note: None
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ da1c029daa85869e16b19a73c228546c98830379: Same failure on other branches
|
livlobo
added
the
stability-period
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ 760a8253ae6478d69da0330133e3efec8e950e4e: Same failure on other branches
|
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ f81f08fc08acd9cd1e0017890d82606741a744f5: Same failure on other branches
|
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ fa6b3adbc5f1e3b85e99a88ca71da11213c8b25a: Same failure on other branches
|
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Aug 30, 2022
…shot Informs cockroachdb#84635 Informs cockroachdb#84162 Fixes cockroachdb#85449. Fixes cockroachdb#83174. This commit considers the `LeaseTransferRejectedBecauseTargetMayNeedSnapshot` status to be a form of a retriable replication change error. It then hooks `Replica.executeAdminCommandWithDescriptor` up to consult this status in its retry loop. This avoids spurious errors when a split gets blocked behind a lateral replica move like we see in the following situation: 1. issue AdminSplit 2. range in joint config, first needs to leave (maybeLeaveAtomicChangeReplicas) 3. to leave, needs to transfer lease from voter_outgoing to voter_incoming 4. can’t transfer lease because doing so is deemed to be potentially unsafe Release note: None Release justification: Low risk.
craig bot
pushed a commit
that referenced
this issue
Aug 31, 2022
85405: kv: retry AdminSplit on LeaseTransferRejectedBecauseTargetMayNeedSnapshot r=shralex a=nvanbenschoten Informs #84635. Informs #84162. Fixes #85449. Fixes #83174. This commit considers the `LeaseTransferRejectedBecauseTargetMayNeedSnapshot` status to be a form of a retriable replication change error. It then hooks `Replica.executeAdminCommandWithDescriptor` up to consult this status in its retry loop. This avoids spurious errors when a split gets blocked behind a lateral replica move like we see in the following situation: 1. issue AdminSplit 2. range in joint config, first needs to leave (maybeLeaveAtomicChangeReplicas) 3. to leave, needs to transfer lease from voter_outgoing to voter_incoming 4. can’t transfer lease because doing so is deemed to be potentially unsafe Release note: None Release justification: Low risk, resolves flaky test. 87137: storage: default to TableFormatPebblev1 in backups r=itsbilal,dt a=jbowens If the v22.2 upgrade has not yet been finalized, so we're not permitted to use the new TableFormatPebblev2 sstable format, default to TableFormatPebblev1 which is the format used by v22.1 internally. This change is intended to allow us to remove code for understanding the old RocksDB table format version sooner (eg, v23.1). Release justification: low-risk updates to existing functionality Release note: None 87152: sql: encode either 0 or 1 spans in scan gists r=mgartner a=mgartner #### dev: add rewritable paths for pkg/sql/opt/exec/explain tests This commit adds fixtures in `pkg/sql/opt/testutils/opttester/testfixtures` as rewritable paths for tests in `pkg/sql/opt/exec/explain`. This prevents `dev test pkg/sql/opt/exec/explain` from erring when the `--rewrite` flag is used. Release justification: This is a test-only change. Release note: None #### sql: encode either 0 or 1 spans in scan gists In plan gists, we no longer encode the exact number of spans for scans so that two queries with the same plan but a different number of spans have the same gist. In addition, plan gists are now decoded with the `OnlyShape` flag which prints any non-zero number of spans as "1+ spans" and removes attributes like "missing stats" from scans. Fixes #87138 Release justification: This is a minor change that makes plan gist instrumentation more scalable. Release note (bug fix): The Explain Tab inside the Statement Details page now groups plans that have the same shape but a different number of spans in corresponding scans. 87154: roachtest: stop cockroach gracefully when upgrading nodes r=yuzefovich a=yuzefovich This commit makes it so that we stop cockroach nodes gracefully when upgrading them. Previous abrupt behavior of stopping the nodes during the upgrade could lead to test flakes because the nodes were not being properly drained. Here is one scenario for how one of the flakes (`pq: version mismatch in flow request: 65; this node accepts 69 through 69`, which means that a gateway running an older version asks another node running a newer version to do DistSQL computation, but the versions are not DistSQL compatible): - we are in a state when node 1 is running a newer version when node 2 is running an older version. Importantly, node 1 was upgraded "abruptly" meaning that it wasn't properly drained; in particular, it didn't send DistSQL draining notification through gossip. - newer node has already been started but its DistSQL server hasn't been started yet (although it already can accept incoming RPCs - see comments on `distsql.ServerImpl.Start` for more details). This means that newer node has **not** sent through gossip an update about its DistSQL version. - node 2 acts as the gateway for a query that reads some data that node 1 is the leaseholder for. During the physical planning, older node 2 checks whether newer node 1 is "healthy and compatible", and node 1 is deemed both healthy (because it can accept incoming RPCs) and is compatible (because node 2 hasn't received updated DistSQL version of node 1 since it hasn't been sent yet). As a result, node 2 plans a read on node 1. - when node 1 receives that request, it errors out with "version mismatch" error. This whole problem is solved if we stop nodes gracefully when upgrading them. In particular, this will mean that node 1 would first dissipate its draining notification across the cluster, so during the physical planning it will only be considered IFF node 1 has already communicated its updated DistSQL version, and then it would be deemed DistSQL-incompatible. I verified that this scenario is possible (with manual adjustments of the version upgrade test and cockroach binary to insert a delay) and that it's fixed by this commit. I believe it is likely that other flake types have the same root cause, but I haven't verified it. Fixes: #87104. Release justification: test-only change. Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Jackson Owens <[email protected]> Co-authored-by: Marcus Gartner <[email protected]> Co-authored-by: Yahor Yuzefovich <[email protected]>
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ 7c8d6e5034b135d475440c8e93385b229dea512f: Same failure on other branches
|
msbutler
changed the title
msbutler
changed the title
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ 052a73d5942c460322afc299aa21ca3d772bf96f: Same failure on other branches
|
|
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ 9eb4da2a351b2fe4706be9aa4af27ee2e60b405e: Same failure on other branches
|
|
@dt looks like the job was in an unexpected failed state again. Unassigning myself and assigning you as L2 for further investigation. |
msbutler
changed the title
|
Here, the job failed because of: |
exalate-issue-sync
bot
changed the title
exalate-issue-sync
bot
removed
the
X-noreuse
|
Fixed by #89611. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.1 @ 7b257ecbb2c0bd9842e33a816b0907ad64a89787:
Help
See: roachtest README
See: How To Investigate (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-17771
The text was updated successfully, but these errors were encountered: