kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1.0 @ f1a130b93c47ddd9a11408d23f48cb6ee96eb38e:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode971182268
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 5964523526688677024
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3807037872
    kvnemesis.go:165: error applying x.AdminMerge(ctx, tk(8392379221266339256)) // kv/kvserver/replica_command.go:807: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]: kv/kvserver/replica_command.go:807: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3807037872/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3807037872/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3807037872/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3807037872/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode971182268
--- FAIL: TestKVNemesisMultiNode (16.06s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvnemesis: TestKVNemesisMultiNode failed #101721 kv/kvnemesis: TestKVNemesisMultiNode failed [C-test-failure O-robot T-kv-replication branch-master]
• kv/kvnemesis: TestKVNemesisMultiNode failed #101048 kv/kvnemesis: TestKVNemesisMultiNode failed [C-test-failure O-robot T-kv branch-release-22.2]
• kv/kvnemesis: TestKVNemesisMultiNode failed #100417 kv/kvnemesis: TestKVNemesisMultiNode failed [C-test-failure O-robot T-kv branch-release-23.1]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-27430

[tbg]

Closing this out since it's on the deprecated 23.1.0 branch.