roachtest: follower-reads/mixed-version/single-region failed

[cockroach-teamcity]

roachtest.follower-reads/mixed-version/single-region failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

(follower_reads.go:871).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 08:55:30-08:55:40: n1 ratio: 0.832 n2 ratio: 0.011 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:55:40-08:55:50: n1 ratio: 1.012 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:55:50-08:56:00: n1 ratio: 0.978 n2 ratio: 0.000 n3 ratio: 0.001 n4 ratio: 0.000 
interval 08:56:00-08:56:10: n1 ratio: 0.430 n2 ratio: 0.002 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:56:10-08:56:20: n1 ratio: 0.906 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:56:20-08:56:30: n1 ratio: 0.990 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:56:30-08:56:40: n1 ratio: 1.008 n2 ratio: 0.005 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:56:40-08:56:50: n1 ratio: 1.004 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:56:50-08:57:00: n1 ratio: 0.816 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:00-08:57:10: n1 ratio: 0.675 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:10-08:57:20: n1 ratio: 0.764 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:20-08:57:30: n1 ratio: 0.527 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:30-08:57:40: n1 ratio: 0.615 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:40-08:57:50: n1 ratio: 1.001 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:57:50-08:58:00: n1 ratio: 0.374 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:00-08:58:10: n1 ratio: 0.530 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:10-08:58:20: n1 ratio: 0.999 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:20-08:58:30: n1 ratio: 0.997 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:30-08:58:40: n1 ratio: 0.778 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:40-08:58:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:58:50-08:59:00: n1 ratio: 0.815 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:59:00-08:59:10: n1 ratio: 0.902 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 08:59:10-08:59:20: n1 ratio: 0.999 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 
(mixedversion.go:694).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: follower-reads/mixed-version/single-region failed #129546 roachtest: follower-reads/mixed-version/single-region failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]
• roachtest: follower-reads/mixed-version/single-region failed #129224 roachtest: follower-reads/mixed-version/single-region failed [C-test-failure O-roachtest O-robot P-1 T-kv X-duplicate branch-release-23.2.10-rc]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-41591

[andrewbaptist]

@renatolabs I haven't looked into this yet, but this run included your commit to disable the test, but still failed. It also says Deployment mode: shared-process in the logs.

[renatolabs]

@andrewbaptist it doesn't include my commit, it's the rc branch for 24.1.4.

Do you mind stamping these rc PRs, please: #129516 and #129558.

[andrewbaptist]

Sorry - I misread the commit history, and got thrown off by the rc branch. Thanks for the quick turnaround!

[renatolabs]

Shared-process disabled on this branch.