roachtest: disk-stalled/cgroup/read-write/logs-too=false failed #103661
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
T-storage
Storage Team
Milestone
Comments
cockroach-teamcity
added
branch-master
jbowens
removed
the
release-blocker
craig bot
pushed a commit
that referenced
this issue
May 19, 2023
103666: multiregionccl: deflake the multi-region cold start test r=JeffSwenson a=JeffSwenson I ran the test 2500 times with no flakes. The core source of the flakes is the test depends on every range having an available replica in every region. Any amount of replica movement causes the test to fail because many regions only have a single replica and temporary unavailability causes the sql servers to pick a replica that is available, but unreachable due to a networking hook injected by the test. There are a few independent changes in this PR: 1. It includes the settings from #100320. The lease transfer setting improves test performance and disabling load based splitting helps avoid replica migration. 2. Based on the insight from #96334 (comment) the test checks the span conformance report for all servers instead of only one of the servers. 3. The test no longer triggers scatters while waiting for the span config bounds to conform. The scatters increased test flakiness and are not required to achieve conformance as we proactively nudge replicas through the replicate queue after #100349. 4. WaitForFullReplication is called to ensure replication queues are empty and there are no pending range moves. Fixes #96334 Release Note: None 103680: roachtest: fix disk-stalled/* roachtests r=jbowens a=jbowens In #103198, I mistakenly constructed a context with a timeout outside the deferral, resulting in the defer'd Unstall always timing out if the remainder of test took longer than a minute. Epic: None Fixes: #103664. Fixes: #103663. Fixes: #103662. Fixes: #103661. Release note: none Co-authored-by: Jeff <[email protected]> Co-authored-by: Jackson Owens <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.disk-stalled/cgroup/read-write/logs-too=false failed with artifacts on master @ 1df2f0e8dd222a21d799c63893fed65cb6a8ba99:
Parameters:
ROACHTEST_cloud=gce,ROACHTEST_cpu=4,ROACHTEST_encrypted=true,ROACHTEST_fs=ext4,ROACHTEST_localSSD=false,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-28116
The text was updated successfully, but these errors were encountered: