kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1 @ 0f97132ad066321e6f65a83fe2098788162445ae:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode57094381
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 6880893433380561686
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3439017642
    kvnemesis.go:165: error applying x.AdminMerge(ctx, tk(14624680467153224630)) // kv/kvserver/replica_command.go:835: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.001s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]: kv/kvserver/replica_command.go:835: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.001s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]
    kvnemesis.go:165: error applying x.AdminMerge(ctx, tk(13806048622469307242)) // kv/kvserver/replica_command.go:835: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.001s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]: kv/kvserver/replica_command.go:835: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.001s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3439017642/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3439017642/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3439017642/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3439017642/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 2
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode57094381
--- FAIL: TestKVNemesisMultiNode (26.86s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-33872

[nvanbenschoten]

I can quickly reproduce this with the following diff:

diff --git a/pkg/kv/kvnemesis/generator.go b/pkg/kv/kvnemesis/generator.go
index b16e8879bb6..5e87c759bc7 100644
--- a/pkg/kv/kvnemesis/generator.go
+++ b/pkg/kv/kvnemesis/generator.go
@@ -401,7 +401,7 @@ func newAllOperationsConfig() GeneratorConfig {
                },
                Merge: MergeConfig{
                        MergeNotSplit: 1,
-                       MergeIsSplit:  1,
+                       MergeIsSplit:  10,
                },
                ChangeReplicas: ChangeReplicasConfig{
                        AddReplica:        1,
diff --git a/pkg/kv/kvserver/replica_command.go b/pkg/kv/kvserver/replica_command.go
index 75ea8ca228c..ed7593a234f 100644
--- a/pkg/kv/kvserver/replica_command.go
+++ b/pkg/kv/kvserver/replica_command.go
@@ -874,7 +874,7 @@ func waitForReplicasInit(
        rangeID roachpb.RangeID,
        replicas []roachpb.ReplicaDescriptor,
 ) error {
-       return timeutil.RunWithTimeout(ctx, "wait for replicas init", 5*time.Second, func(ctx context.Context) error {
+       return timeutil.RunWithTimeout(ctx, "wait for replicas init", 1*time.Millisecond, func(ctx context.Context) error {
                g := ctxgroup.WithContext(ctx)
                for _, repl := range replicas {
                        repl := repl // copy for goroutine

when running:

dev test pkg/kv/kvnemesis -f=TestKVNemesisMultiNode --stress

This is a timing flake which we've seen in the past on kvnemesis. It's not concerning, but we should make it not flaky.