roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed

[cockroach-teamcity]

roachtest.follower-reads/mixed-version/survival=region/locality=global/reads=strong failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 19 intervals > 4 threshold. Bad intervals:
interval 09:14:00-09:14:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.986 n5 ratio: 1.003 n6 ratio: 1.000 
interval 09:14:10-09:14:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 0.986 n6 ratio: 0.985 
interval 09:14:20-09:14:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:14:30-09:14:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:14:40-09:14:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:14:50-09:15:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:15:00-09:15:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.003 n6 ratio: 1.000 
interval 09:15:10-09:15:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:15:20-09:15:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.003 n6 ratio: 1.000 
interval 09:15:30-09:15:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 0.997 n6 ratio: 1.000 
interval 09:15:40-09:15:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:15:50-09:16:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:16:00-09:16:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:16:10-09:16:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.000 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:16:20-09:16:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.003 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:16:30-09:16:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.998 n5 ratio: 1.000 n6 ratio: 1.000 
interval 09:16:40-09:16:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.158 n5 ratio: 1.757 n6 ratio: 1.000 
interval 09:16:50-09:17:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.000 n5 ratio: 2.010 n6 ratio: 0.985 
interval 09:17:00-09:17:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 0.002 n5 ratio: 2.054 n6 ratio: 1.000 
(mixedversion.go:695).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/survival=region/locality=global/reads=strong/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed [VM disappeared from roachprod cache] #129394 roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed [VM disappeared from roachprod cache] [C-test-failure O-roachtest O-robot T-testeng X-noreuse branch-release-24.2.0-rc]
• roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed #129167 roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed [C-test-failure O-roachtest O-robot P-1 T-kv branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-41667

[renatolabs]

#129604

[renatolabs]

Shared-process deployments disabled on this branch.