roachtest: failover/partial/lease-liveness failed [azure]

[cockroach-teamcity]

roachtest.failover/partial/lease-liveness failed with artifacts on master @ e90d15c9807775a1a1d89f75cac4c9383c325709:

(failover.go:1769).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2351).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-liveness/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-38508

[nvanbenschoten]

On n1, we see a disk stall:

disk stall detected: disk slowness detected: syncdata on file 001255.log has been ongoing for 21.1s

On n5, we see a context cancellation while clearing iptables, which may be related to the monitor noticing the dead node:

teamcity-15141267-1715060891-96-n8cpu2:[1 2 3 4 5 6 7 8]: sudo iptables -F
   5: 	<err> context canceled

This is on azure, which is suspect:

ROACHTEST_cloud=azure

I'll bucket this Azure disk stall as an infra-flake on Azure and hand off to test-eng for monitoring.

[blathers-crl]

cc @cockroachdb/test-eng

[renatolabs]

Referencing #97968 for us to keep track of this; closing this issue.