roachtest: cdc/poller/rangefeed=false failed [cannot recover PENDING transaction in same epoch] #62064
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
GA-blocker
O-roachtest
O-robot
Originated from a bot.
Comments
cockroach-teamcity
added
branch-master
|
Similar to #61992 |
tbg
changed the title
nvanbenschoten
added
GA-blocker
and removed
release-blocker
craig bot
pushed a commit
that referenced
this issue
Mar 29, 2021
62705: workload/schemachange: check enum's non-public status during ADD REGION r=ajwerner a=otan Previously, having two ADD REGIONS in the same transaction with the same value would error "unexpectedly" by the workload changer as we did not inject the expected error in. This is because the region on the first ADD REGION is not immediately made a public enum member, and does not show up on SHOW REGIONS on the second generation of ADD REGION. To fix this, we now detect whether the enum exists as a non-public member and if so add the expected error code. Also re-enables ADD REGION operation generation. Release note: None 62744: roachtest: remove cdc/poller/rangefeed=false r=nvanbenschoten a=nvanbenschoten Closes #62064. This isn't fixed, but consolidating to #61992. This should have been removed in 8a81eac. The test is now effectively identical to `cdc/tpcc-1000`, just less aggressive in its assertions. Co-authored-by: Oliver Tan <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Mar 29, 2021
Fixes cockroachdb#61992. Fixes cockroachdb#62064. This commit fixes a bug uncovered recently (for less than obvious reasons) in cdc roachtests where a STAGING transaction could have its transaction record moved back to a PENDING state without changing epochs but after its timestamp was bumped. This could result in concurrent transaction recovery attempts returning `programming error: cannot recover PENDING transaction in same epoch` errors, because such a state transition was not expected to be possible by transaction recovery. However, as we found in cockroachdb#61992, this has actually been possible since 01bc20e. This commit fixes the bug by detecting cases where a pusher knows of a failed parallel commit and selectively upgrading PUSH_TIMESTAMP push attempts to PUSH_ABORTs. This has no effect on pushes that fail with a TransactionPushError. Such pushes will still wait on the pushee to retry its commit and eventually commit or abort. It also has no effect on expired pushees, as they would have been aborted anyway. This only impacts pushes which would have succeeded due to priority mismatches. In these cases, the push acts the same as a short-circuited transaction recovery process, because the transaction recovery procedure always finalizes target transactions, even if initiated by a PUSH_TIMESTAMP. This seems very rare in practice, as it requires a few specific interactions to line up just right, including: - a STAGING transaction that has one of its in-flight intent writes bumped - a rangefeed processor listening to that intent write - a separate request that conflicts with a different intent - a STAGING transaction which expires to allow transaction recovery - a rangefeed processor push between the time of the request push and the request recovery Still, this fix well contained, so I think we should backport it to all of the release branches. However, since this issue does seem rare and also can not cause corruption or atomicity violations, I wanted to be conservative with the backport, so I'm going to let this bake on master + release-21.1 for a few weeks before merging the backport. Release notes (bug fix): an improper interaction between conflicting transactions which could result in spurious `cannot recover PENDING transaction in same epoch` errors was fixed.
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Mar 30, 2021
Fixes cockroachdb#61992. Fixes cockroachdb#62064. This commit fixes a bug uncovered recently (for less than obvious reasons) in cdc roachtests where a STAGING transaction could have its transaction record moved back to a PENDING state without changing epochs but after its timestamp was bumped. This could result in concurrent transaction recovery attempts returning `programming error: cannot recover PENDING transaction in same epoch` errors, because such a state transition was not expected to be possible by transaction recovery. However, as we found in cockroachdb#61992, this has actually been possible since 01bc20e. This commit fixes the bug by detecting cases where a pusher knows of a failed parallel commit and selectively upgrading PUSH_TIMESTAMP push attempts to PUSH_ABORTs. This has no effect on pushes that fail with a TransactionPushError. Such pushes will still wait on the pushee to retry its commit and eventually commit or abort. It also has no effect on expired pushees, as they would have been aborted anyway. This only impacts pushes which would have succeeded due to priority mismatches. In these cases, the push acts the same as a short-circuited transaction recovery process, because the transaction recovery procedure always finalizes target transactions, even if initiated by a PUSH_TIMESTAMP. This seems very rare in practice, as it requires a few specific interactions to line up just right, including: - a STAGING transaction that has one of its in-flight intent writes bumped - a rangefeed processor listening to that intent write - a separate request that conflicts with a different intent - a STAGING transaction which expires to allow transaction recovery - a rangefeed processor push between the time of the request push and the request recovery Still, this fix well contained, so I think we should backport it to all of the release branches. However, since this issue does seem rare and also can not cause corruption or atomicity violations, I wanted to be conservative with the backport, so I'm going to let this bake on master + release-21.1 for a few weeks before merging the backport. Release notes (bug fix): an improper interaction between conflicting transactions which could result in spurious `cannot recover PENDING transaction in same epoch` errors was fixed.
craig bot
pushed a commit
that referenced
this issue
Mar 30, 2021
62761: kv: prevent STAGING -> PENDING transition during high-priority push r=nvanbenschoten a=nvanbenschoten Fixes #61992. Fixes #62064. This commit fixes a bug uncovered recently (for less than obvious reasons) in cdc roachtests where a STAGING transaction could have its transaction record moved back to a PENDING state without changing epochs but after its timestamp was bumped. This could result in concurrent transaction recovery attempts returning `programming error: cannot recover PENDING transaction in same epoch` errors, because such a state transition was not expected to be possible by transaction recovery. However, as we found in #61992, this has actually been possible since 01bc20e. This commit fixes the bug by detecting cases where a pusher knows of a failed parallel commit and selectively upgrading PUSH_TIMESTAMP push attempts to PUSH_ABORTs. This has no effect on pushes that fail with a TransactionPushError. Such pushes will still wait on the pushee to retry its commit and eventually commit or abort. It also has no effect on expired pushees, as they would have been aborted anyway. This only impacts pushes which would have succeeded due to priority mismatches. In these cases, the push acts the same as a short-circuited transaction recovery process, because the transaction recovery procedure always finalizes target transactions, even if initiated by a PUSH_TIMESTAMP. This seems very rare in practice, as it requires a few specific interactions to line up just right, including: - a STAGING transaction that has one of its in-flight intent writes bumped - a rangefeed processor listening to that intent write - a separate request that conflicts with a different intent - a STAGING transaction which expires to allow transaction recovery - a rangefeed processor push between the time of the request push and the request recovery Still, this fix well contained, so I think we should backport it to all of the release branches. However, since this issue does seem rare and also can not cause corruption or atomicity violations, I wanted to be conservative with the backport, so I'm going to let this bake on master + release-21.1 for a few weeks before merging the backport. Release notes (bug fix): an improper interaction between conflicting transactions which could result in spurious `cannot recover PENDING transaction in same epoch` errors was fixed. Co-authored-by: Nathan VanBenschoten <[email protected]>
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Mar 30, 2021
Fixes cockroachdb#61992. Fixes cockroachdb#62064. This commit fixes a bug uncovered recently (for less than obvious reasons) in cdc roachtests where a STAGING transaction could have its transaction record moved back to a PENDING state without changing epochs but after its timestamp was bumped. This could result in concurrent transaction recovery attempts returning `programming error: cannot recover PENDING transaction in same epoch` errors, because such a state transition was not expected to be possible by transaction recovery. However, as we found in cockroachdb#61992, this has actually been possible since 01bc20e. This commit fixes the bug by detecting cases where a pusher knows of a failed parallel commit and selectively upgrading PUSH_TIMESTAMP push attempts to PUSH_ABORTs. This has no effect on pushes that fail with a TransactionPushError. Such pushes will still wait on the pushee to retry its commit and eventually commit or abort. It also has no effect on expired pushees, as they would have been aborted anyway. This only impacts pushes which would have succeeded due to priority mismatches. In these cases, the push acts the same as a short-circuited transaction recovery process, because the transaction recovery procedure always finalizes target transactions, even if initiated by a PUSH_TIMESTAMP. This seems very rare in practice, as it requires a few specific interactions to line up just right, including: - a STAGING transaction that has one of its in-flight intent writes bumped - a rangefeed processor listening to that intent write - a separate request that conflicts with a different intent - a STAGING transaction which expires to allow transaction recovery - a rangefeed processor push between the time of the request push and the request recovery Still, this fix well contained, so I think we should backport it to all of the release branches. However, since this issue does seem rare and also can not cause corruption or atomicity violations, I wanted to be conservative with the backport, so I'm going to let this bake on master + release-21.1 for a few weeks before merging the backport. Release notes (bug fix): an improper interaction between conflicting transactions which could result in spurious `cannot recover PENDING transaction in same epoch` errors was fixed.
nvanbenschoten
added a commit
to nvanbenschoten/cockroach
that referenced
this issue
Apr 1, 2021
Closes cockroachdb#62064. This isn't fixed, but consolidating to cockroachdb#61992. This should have been removed in 8a81eac. The test is now effectively identical to `cdc/tpcc-1000`, just less aggressive in its assertions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(roachtest).cdc/poller/rangefeed=false failed on master@e9387a6e5dfdad71c74ccd0a07c907632613fa3e:
More
Artifacts: /cdc/poller/rangefeed=false
See this test on roachdash
powered by pkg/cmd/internal/issues
The text was updated successfully, but these errors were encountered: