roachtest: clearrange/zfs/checks=true failed #108726
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
T-storage
Storage Team
Milestone
Comments
cockroach-teamcity
added
branch-master
|
n5 crashed due to a replica inconsistency in system bytes: |
|
Possibly related to #93896? |
|
Based on a conversation with the KV team on internal slack, stats deltas in sysbytes are normal and expected due to non-determinism and races with lease acquisitions. Since clearrange is the only roachtest that asserts on MVCC stats mismatches anywhere in CockroachDB, I'll update this roachtest to not fail on stats mismatches for now. |
itsbilal
removed
the
release-blocker
|
@jbowens yep it is. For some reason it also didn't notify me of your comment until I sent mine 😬 something is weirdly messed up about just this issue and Github notifications for me. |
craig bot
pushed a commit
that referenced
this issue
Aug 16, 2023
108673: kvstreamer: minor cleanup and unification r=yuzefovich a=yuzefovich This commit performs some minor cleanup to unify the code a bit between GetResp and ScanResp. The only change that is not a noop is the fact that we're now `nil`ing out `ResumeSpan` on GetResponses as well as on ScanResponses when SingleRowLookup hint is `false`. Originally, we were unsetting it for all ScanResponses but in 6343df3 this was lost making the behavior different based on the hint; GetResponses never had this `nil`ing out in the first place. The rationale for actually unsetting the ResumeSpan for both types of the responses is somewhat weak (not confusing the Streamer's user (which doesn't actually inspect the ResumeSpan field) as well as to allow for GC of the keys sooner), but it's better for this behavior to be unified. Epic: None Release note: None 108786: roachtest: allow stats mismatches in clearrange roachtest r=RaduBerinde a=itsbilal Previously, the clearrange roachtest was the only place anywhere in the CockroachDB codebase where we would assert on MVCC stats matching between replicas. This would trip up and fail the clearrange roachtest even in known cases of MVCC stats mismatches. This change removes the code to assert on stats mismatches with consistency checks, but retains the clearrange roachtest's use of aggressive consistency checks, so mismatches in checksums computed on data in each replica will continue to fatal the test. Related to #93896. Fixes #108726. Epic: none Release note: None Co-authored-by: Yahor Yuzefovich <[email protected]> Co-authored-by: Bilal Akhtar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.clearrange/zfs/checks=true failed with artifacts on master @ c13bf7633cbb416d9e43f8c57b1e309fab1110ce:
Parameters:
ROACHTEST_arch=amd64,ROACHTEST_cloud=gce,ROACHTEST_cpu=16,ROACHTEST_encrypted=true,ROACHTEST_fs=zfs,ROACHTEST_localSSD=true,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
See: Grafana
This test on roachdash | Improve this report!
Jira issue: CRDB-30614
The text was updated successfully, but these errors were encountered: