roachtest: backup/mvcc-range-tombstones failed #97694
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
release-blocker
Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
T-disaster-recovery
Milestone
Comments
cockroach-teamcity
added
branch-master
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 13c58f621519794e775b7cfc4d8b557bc99eeca0: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ b0e5507f74c07e13cfda8cda8b9079b457a9f37d: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 21786aa112e6b822858f281c1cc59608987c5c0a: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ cf14ad694ee562676f53e36fa8495206c3aed61f: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 14b43be03c1c246765be17aa15245493a42c401d: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 43a25a09f92408accdeb3cfed1eb35e02d375877: Parameters: |
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 1b162d1b274eec7b307fbbfca7294460bfdef025: Parameters: |
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this issue
Mar 9, 2023
In cockroachdb#96691, we changed the return type of mvccExportToWriter such that it now indicates when a CPU limit has been reached. As part of that change, when the CPU limit was reached, we would immedately `return` rather than `break`ing out of the loop. This introduced a bug, since the code after the loop that the `break` was taking us to is important. Namely, we may have previously buffered range keys that we need to write into our response still. By replacing the break with a return, these range keys were lost. The end-user impact of this is that a BACKUP that _ought_ to have included range keys (such as a backup of a table with a rolled back IMPORT) would not include those range keys and thus would end up resurecting deleted keys upon restore. This PR brings back the `break` and adds a test that covers exporting with range keys under CPU exhaustion. This bug only ever existed on master. Informs cockroachdb#97694 Epic: none Release note: None
|
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b: Parameters: |
craig bot
pushed a commit
that referenced
this issue
Mar 9, 2023
97717: multitenant: add AdminUnsplitRequest capability r=knz a=ecwall Fixes #97716 1) Add a new `tenantcapabilitiespb.CanAdminUnsplit` capability. 2) Check capability in `Authorizer.HasCapabilityForBatch`. 3) Remove `ExecutorConfig.RequireSystemTenant` call from `execFactory.ConstructAlterTableUnsplit`, `execFactory.ConstructAlterTableUnsplitAll`. Release note: None 98250: kvserver: add minimum cpu lb split threshold r=andrewbaptist a=kvoli Previously, `kv.range_split.load_cpu_threshold` had no minimum setting value. It is undesirable to allow users to set this setting to low as excessive splitting may occur. `kv.range_split.load_cpu_threshold` now has a minimum setting value of `10ms`. See #96869 for additional context on the threshold. Resolves: #98107 Release note (ops change): `kv.range_split.load_cpu_threshold` now has a minimum setting value of `10ms`. 98270: dashboards: add replica cpu to repl dashboard r=xinhaoz a=kvoli In #96127 we added the option to load balance replica CPU instead of QPS across stores in a cluster. It is desirable to view the signal being controlled for rebalancing in the replication dashboard, similar to QPS. This pr adds the `rebalancing.cpunanospersecond` metric to the replication metrics dashboard. The avg QPS graph on the replication graph previously described the metric as "Exponentially weighted average", however this is not true. This pr updates the description to just be "moving average" which is accurate. Note that follow the workload does use an exponentially weighted value, however the metric in the dashboard is not the same. This pr also updates the graph header to include Replica in the title: "Average Replica Queries per Node". QPS is specific to replicas. This is already mentioned in the description. Resolves: #98109 98289: storage: mvccExportToWriter should always return buffered range keys r=adityamaru a=stevendanna In #96691, we changed the return type of mvccExportToWriter such that it now indicates when a CPU limit has been reached. As part of that change, when the CPU limit was reached, we would immedately `return` rather than `break`ing out of the loop. This introduced a bug, since the code after the loop that the `break` was taking us to is important. Namely, we may have previously buffered range keys that we need to write into our response still. By replacing the break with a return, these range keys were lost. The end-user impact of this is that a BACKUP that _ought_ to have included range keys (such as a backup of a table with a rolled back IMPORT) would not include those range keys and thus would end up resurecting deleted keys upon restore. This PR brings back the `break` and adds a test that covers exporting with range keys under CPU exhaustion. This bug only ever existed on master. Informs #97694 Epic: none Release note: None 98329: sql: fix iteration conditions in crdb_internal.scan r=ajwerner a=stevendanna Rather than using the Next() key of the last key in the response when iterating, we should use the resume span. The previous code could result in a failure in the rare case that the end key of our scan exactly matched the successor key of the very last key in the iteration. Epic: none Release note: None Co-authored-by: Evan Wall <[email protected]> Co-authored-by: Austen McClernon <[email protected]> Co-authored-by: Steven Danna <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.backup/mvcc-range-tombstones failed with artifacts on master @ 39c06b5a438c01c93ffbfeeefe702d3f9b620eaf:
Parameters:
ROACHTEST_cloud=gce,ROACHTEST_cpu=8,ROACHTEST_encrypted=false,ROACHTEST_fs=ext4,ROACHTEST_localSSD=true,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-24827
Epic CRDB-21942
The text was updated successfully, but these errors were encountered: