roachtest: kv/quiescence/nodes=3 failed

[cockroach-teamcity]

roachtest.kv/quiescence/nodes=3 failed with artifacts on release-23.1 @ d76049d66d6e3a2cd398d0347a63dd29cc9b928a:

test artifacts and logs in: /artifacts/kv/quiescence/nodes=3/run_1
(test_runner.go:1150).func1: 1 dead node(s) detected

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-28784

[kvoli]

Test succeeds but then fails the dead node detector:

06:29:07 kv.go:461: QPS went from 2968.85 to 2921 with one node down
06:29:07 test_runner.go:1066: tearing down after success; see teardown.log

The test explicitly kills a node but the teardown process doesn't seem to care and considers this a failure.

Same situation as #102054 which is supposed to be resolved by #102162. We still are hitting this though.

Removing release blocker for now.

[erikgrinaker]

I don't think #102162 was backported to 23.1 -- and if it were, maybe these tests weren't opted out. That would also explain the failover disk stall test failures on 23.1: #104960 and #104961.

[blathers-crl]

cc @cockroachdb/test-eng

[kvoli]

Yup that looks like it, I don't see the skip teardown option in 23.1

cockroach/pkg/cmd/roachtest/tests/kv.go

Lines 393 to 395 in 6235182

	Cluster: r.MakeClusterSpec(4),
	Leases: registry.MetamorphicLeases,
	Run: func(ctx context.Context, t test.Test, c cluster.Cluster) {

Where I do see it on master

cockroach/pkg/cmd/roachtest/tests/kv.go

Line 420 in fdf04ff

SkipPostValidations: registry.PostValidationNoDeadNodes,

Re-assigning to test-eng @smg260 - I think the commit just needs a backport to 23.1, however I'm unsure why this only started failing again now.

[smg260]

This failed today because, several roachtest changes were backported via #104882 yesterday, including one which fixed the dead node detector. We missed the subsequent PR (mentioned above) which allowed for a test to skip the "no dead node" assertion.

#104977 should resolve this.