roachtest: expand WAL failover test assertions #121367
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jbowens
requested review from
sumeerbhola,
DarrylWong and
renatolabs
and removed request for
a team
|
This change is |
sumeerbhola
approved these changes
Mar 29, 2024
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @DarrylWong, @jbowens, and @renatolabs)
pkg/cmd/roachtest/tests/disk_stall.go line 184 at r1 (raw file):
if durInFailover < 60*time.Second { t.Errorf("expected s1 to spend at least 60s writing to secondary, but spent %s", durInFailover) }
obvious question: do both assertions fail if failover is not configured?
Update the WAL failover disk stall roachtest to assert that the stalled store does failover to the secondary and that SQL tail latencies remain bounded. Epic: none Release note: none
jbowens
force-pushed
the
diskstall-asserts
branch
from
aa8b303
to
6685ecb
Compare
jbowens
commented
Apr 1, 2024
There was a problem hiding this comment.
TFTR!
bors r=sumeerbhola
Reviewable status:
pkg/cmd/roachtest/tests/disk_stall.go line 184 at r1 (raw file):
Previously, sumeerbhola wrote…
obvious question: do both assertions fail if failover is not configured?
Yeah, it does
19:45:05 test_runner.go:1055: [w0] --- FAIL: disk-stalled/wal-failover/among-stores (3976.90s)
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.013578081s at 2024-04-01T18:49:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.037223403s at 2024-04-01T18:59:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.012550625s at 2024-04-01T19:09:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.013074966s at 2024-04-01T19:19:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.014138031s at 2024-04-01T19:29:00Z
(disk_stall.go:183).runDiskStalledWALFailover: expected s1 to spend at least 60s writing to secondary, but spent 0s
(cluster.go:2344).Run: context canceled
test artifacts and logs in: artifacts/disk-stalled/wal-failover/among-stores/run_1
--- FAIL: disk-stalled/wal-failover/among-stores (3976.90s)
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.013578081s at 2024-04-01T18:49:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.037223403s at 2024-04-01T18:59:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.012550625s at 2024-04-01T19:09:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.013074966s at 2024-04-01T19:19:00Z
(disk_stall.go:174).runDiskStalledWALFailover: unexpectedly high p99.99 latency 1.014138031s at 2024-04-01T19:29:00Z
(disk_stall.go:183).runDiskStalledWALFailover: expected s1 to spend at least 60s writing to secondary, but spent 0s
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the WAL failover disk stall roachtest to assert that the stalled store does failover to the secondary and that SQL tail latencies remain bounded.
Epic: none
Release note: none