kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1 @ 1a5dc2aad16ca4185afcda75d923804324dcfeb9:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1082590582
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 1281984876317631820
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3950191080
    kvnemesis.go:165: error applying x.ScanForUpdate(ctx, tk(5364256596085298844), tk(9743500752805155333), 0) // WriteTooOldError: write for key /Table/100/"5943595007238dc8" at timestamp 1686132671.535140016,0 too old; wrote at 1686132671.580430305,1: {<nil> 0 {0xc00896e6e0} <nil> 1686132671.654509064,0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3950191080/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3950191080/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3950191080/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3950191080/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1082590582
--- FAIL: TestKVNemesisMultiNode (19.52s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvnemesis: TestKVNemesisMultiNode failed #101721 kv/kvnemesis: TestKVNemesisMultiNode failed [C-bug C-test-failure O-robot T-kv-replication branch-master]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-28548

[irfansharif]

23.1 dup of this unfixed master issue: #101721.

[aliher1911]

It is hard to say if it is dup. TestKVNemesisMultiNode is a catch all for all sort of underlying issues that nemesis detects.
I don't see this particular error in any of the failures in the linked issue.
Unfortunately we keep losing artifacts for failures and they are hard to reproduce.

[pav-kv]

@aliher1911 @irfansharif Do any of you still have the artifacts? Otherwise I'm tempted to close this as unactionable, and hope we'll catch it while fixing #101721.

[blathers-crl]

cc @cockroachdb/replication

[irfansharif]

I don't, closing it seems appropriate. I know we bounced a lot of TC agents recently so maybe that's why we lost them? The last failure was only a week ago.

[nvanbenschoten]

I think we're seeing the effect of a non-transactional locking read request that hits write-write contention. We must not handle that correctly.

[tbg]

Possibly related to #105330

[nvanbenschoten]

I've stressed this for over 1.3M iterations and 30 hours on 20 gce nodes without seeing a failure. During that time, I had applied the following patch to increase the number of ScanForUpdate operations in the workload mix:

diff --git a/pkg/kv/kvnemesis/generator.go b/pkg/kv/kvnemesis/generator.go
index 691cca96bcc..d41e760a573 100644
--- a/pkg/kv/kvnemesis/generator.go
+++ b/pkg/kv/kvnemesis/generator.go
@@ -188,14 +188,14 @@ func newAllOperationsConfig() GeneratorConfig {
                GetMissingForUpdate:       1,
                GetExisting:               1,
                GetExistingForUpdate:      1,
-               PutMissing:                1,
-               PutExisting:               1,
+               PutMissing:                25,
+               PutExisting:               25,
                Scan:                      1,
-               ScanForUpdate:             1,
+               ScanForUpdate:             50,
                ReverseScan:               1,
                ReverseScanForUpdate:      1,
-               DeleteMissing:             1,
-               DeleteExisting:            1,
+               DeleteMissing:             25,
+               DeleteExisting:            25,
                DeleteRange:               1,
                DeleteRangeUsingTombstone: 1,
                AddSSTable:                1,

Given that this is an unexpected but innocuous error for a non-transactional ScanForUpdate request (something we never issue in practice), I'm going to close this out. We've already spent $412.85 (according to roachprod) trying to reproduce and it doesn't seem worthwhile to push any harder unless we see it again.