roachtest: restore/nodeShutdown/worker failed #88667
Comments
cockroach-teamcity
added
branch-master
|
roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 51c8aae748d338549400c047796c6c9b892527da: Parameters: Same failure on other branches
|
|
Here is another one stuck on AdminSplit. However, with the new logging added because of #87837 we can see the reason: Note that n3 in this test has been killed to simulate a worker failure. |
|
The We see that the split is stuck on a previous joint config, which is trying to transfer away the range lease to the incoming voter (logic added in the previous release). However, the transfer is being rejected because the incoming voter may need a snapshot. Why can't the transferrer determine whether the incoming voter needs a snapshot? Because the voter_incoming has been killed by the test. So this is quite an interesting situation. In the past, we would have transferred the lease to the dead node, which would have eventually expired, and then allowed another replica in the incoming side of the joint config to acquire the lease. We now get stuck because we refuse to perform a potentially unsafe lease transfer. In fact, this safety check is not even a false positive. The lease transfer would be unsafe — the target is dead! But then we don't have a way out of this situation, which seems quite bad. So the joint config gets stuck. What can we do here? Some options are:
cc. @shralex. I think option 1 might actually be the best here. We've seen fallout from adding these safety checks to the lease transfer performed during a joint config. While it's well-intentioned, it's also hard to justify because we know that the incoming voter was just sent a snapshot. |
This comment was marked as outdated.
This comment was marked as outdated.
A concern with this approach is that the lease transferee needs the be the replica that completes the joint configuration. If we blindly allow lease transfers to dead nodes, then it's possible for the lease transfer to complete, the lease to expire, and then the previous leaseholder to once again acquire the lease (permitted as of #83686). This could then repeat indefinitely without any other replica acquiring the lease and completing the joint configuration. To guard against that, it does feel like we'll need some variant of option 2. Note that this isn't a new issue — I think we can already hit this retry loop in v22.1. |
|
It really feels like what we need is a way for the current leaseholder to invalidate it's current lease, perhaps by issuing a synthetic lease transfer to a non-existing replica (or something of the sort), and amending the logic modified in #83686 to guarantee that only the other (non So the rough idea would be something like:
|
nvanbenschoten
added
GA-blocker
and removed
release-blocker
|
roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 84384b50c023dd4c05fff76af85a6975f5d2b0ab: Parameters: Same failure on other branches
|
|
@aayushshah15, @shralex, and I met last week to discuss this issue. In the short term, we decided that option 1 from #88667 (comment) is the safest choice. I made that change and tested it out on this test. Without any changes, this test has a failure rate of about 7% and gets stuck in a However, it still has a 5% failure rate with a different failure mode. The issue is that even if we allow the lease transfer to go through to the dead node, the VOTER_DEMOTING is still allowed to re-acquire the lease. As a result, the test fails in the same way we saw in #85879, with the job hitting a What we need to step 4 from #85879 (comment) (the one remaining step from that list). Once the VOTER_DEMOTING transfers away the lease, it shouldn't re-acquire it. I made that change as well and found that the failure rate dropped to 0%. To summarize, we need the following two changes:
|
exalate-issue-sync
bot
added
blocks-22.2.0-beta.2
release-blocker
This PR restricts the case when a VOTER_DEMOTING_LEARNER can aquire the lease in a joint configuration to only the case where it was the last leaseholder. Since it is being removed, we only want it to get the lease if no other replica can aquire it, the scenario described in #83687 This fix solves a potential starvation scenario where a VOTER_DEMOTING_LEARNER keeps transferring the lease to the VOTER_INCOMING, succeeding, but then re-acquiring because the VOTER_INCOMING is dead and the lease expires. In this case, we would want another replica to pick up the lease, which would allow us to exit the joint configuration. This PR also removes the leaseHolderRemovalAllowed parameter of CheckCanReceiveLease, since it is always true since 22.2. Release note (bug fix): narrows down the conditions under which a VOTER_DEMOTING_LEARNER can acquire the lease in a joint configuration to a) there has to be an VOTER_INCOMING in the configuration and b) the VOTER_DEMOTING_LEARNER was the last leaseholder. This prevents it from acquiring the lease unless it is the only one that can acquire it, because transferring the lease away is necessary before exiting the joint config (without the fix the system can be stuck in a joint configuration in some rare situations). Fixes: #88667 See also #89340
This PR restricts the case when a VOTER_DEMOTING_LEARNER can aquire the lease in a joint configuration to only the case where it was the last leaseholder. Since it is being removed, we only want it to get the lease if no other replica can aquire it, the scenario described in #83687 This fix solves a potential starvation scenario where a VOTER_DEMOTING_LEARNER keeps transferring the lease to the VOTER_INCOMING, succeeding, but then re-acquiring because the VOTER_INCOMING is dead and the lease expires. In this case, we would want another replica to pick up the lease, which would allow us to exit the joint configuration. This PR also removes the leaseHolderRemovalAllowed parameter of CheckCanReceiveLease, since it is always true since 22.2. Release note (bug fix): narrows down the conditions under which a VOTER_DEMOTING_LEARNER can acquire the lease in a joint configuration to a) there has to be an VOTER_INCOMING in the configuration and b) the VOTER_DEMOTING_LEARNER was the last leaseholder. This prevents it from acquiring the lease unless it is the only one that can acquire it, because transferring the lease away is necessary before exiting the joint config (without the fix the system can be stuck in a joint configuration in some rare situations). Fixes: #88667 See also #89340
|
We still need to land #89340 to resolve this. |
This PR restricts the case when a VOTER_DEMOTING_LEARNER can aquire the lease in a joint configuration to only the case where it was the last leaseholder. Since it is being removed, we only want it to get the lease if no other replica can aquire it, the scenario described in cockroachdb#83687 This fix solves a potential starvation scenario where a VOTER_DEMOTING_LEARNER keeps transferring the lease to the VOTER_INCOMING, succeeding, but then re-acquiring because the VOTER_INCOMING is dead and the lease expires. In this case, we would want another replica to pick up the lease, which would allow us to exit the joint configuration. Release note (bug fix): narrows down the conditions under which a VOTER_DEMOTING_LEARNER can acquire the lease in a joint configuration to a) there has to be an VOTER_INCOMING in the configuration and b) the VOTER_DEMOTING_LEARNER was the last leaseholder. This prevents it from acquiring the lease unless it is the only one that can acquire it, because transferring the lease away is necessary before exiting the joint config (without the fix the system can be stuck in a joint configuration in some rare situations). Release justification: solves a serious bug. Fixes: cockroachdb#88667 See also cockroachdb#89340
89340: kv: bypass lease transfer safety checks during joint consensus r=shralex a=nvanbenschoten This commit adds logic to bypass lease transfer safety checks (added in 034611b) when in a joint configuration and transferring the lease from a VOTER_DEMOTING to a VOTER_INCOMING. We do so because we could get stuck without a path to exit the joint configuration if we rejected this lease transfer while waiting to confirm that the target is up-to-date on its log. That confirmation may never arrive if the target is dead or partitioned away, and while we'd rather not transfer the lease to a dead node, at least we have a mechanism to recovery from that state. We also just sent the VOTER_INCOMING a snapshot (as a LEARNER, before promotion), so it is unlikely that the replica is actually dead or behind on its log. A better alternative here would be to introduce a mechanism to choose an alternate lease transfer target after some amount of time, if the lease transfer to the VOTER_INCOMING cannot be confirmed to be safe. We may do this in the future, but given the proximity to the release and given that this matches the behavior in v22.1, we choose this approach for now. Release note: None Release justification: Needed to resolve release blocker. Fixes: #88667 See also #89564 Co-authored-by: Nathan VanBenschoten <[email protected]>
Fixes cockroachdb#88667. This commit adds logic to bypass lease transfer safety checks (added in 034611b) when in a joint configuration and transferring the lease from a VOTER_DEMOTING to a VOTER_INCOMING. We do so because we could get stuck without a path to exit the joint configuration if we rejected this lease transfer while waiting to confirm that the target is up-to-date on its log. That confirmation may never arrive if the target is dead or partitioned away, and while we'd rather not transfer the lease to a dead node, at least we have a mechanism to recovery from that state. We also just sent the VOTER_INCOMING a snapshot (as a LEARNER, before promotion), so it is unlikely that the replica is actually dead or behind on its log. A better alternative here would be to introduce a mechanism to choose an alternate lease transfer target after some amount of time, if the lease transfer to the VOTER_INCOMING cannot be confirmed to be safe. We may do this in the future, but given the proximity to the release and given that this matches the behavior in v22.1, we choose this approach for now. Release note: None Release justification: Needed to resolve release blocker.
Fixes cockroachdb#88667. This commit adds logic to bypass lease transfer safety checks (added in 034611b) when in a joint configuration and transferring the lease from a VOTER_DEMOTING to a VOTER_INCOMING. We do so because we could get stuck without a path to exit the joint configuration if we rejected this lease transfer while waiting to confirm that the target is up-to-date on its log. That confirmation may never arrive if the target is dead or partitioned away, and while we'd rather not transfer the lease to a dead node, at least we have a mechanism to recovery from that state. We also just sent the VOTER_INCOMING a snapshot (as a LEARNER, before promotion), so it is unlikely that the replica is actually dead or behind on its log. A better alternative here would be to introduce a mechanism to choose an alternate lease transfer target after some amount of time, if the lease transfer to the VOTER_INCOMING cannot be confirmed to be safe. We may do this in the future, but given the proximity to the release and given that this matches the behavior in v22.1, we choose this approach for now. Release note: None Release justification: Needed to resolve release blocker.
lunevalex
added
the
C-bug
roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 89f4ad907a1756551bd6864c3e8516eeff6b0e0a:
Parameters:
ROACHTEST_cloud=gce,ROACHTEST_cpu=4,ROACHTEST_encrypted=false,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-19950
The text was updated successfully, but these errors were encountered: