kv/kvnemesis: TestKVNemesisMultiNode failed #100417

cockroach-teamcity · 2023-04-02T10:03:38Z

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1 @ 63b683eab3e0ebfff812bb54fdac936c101d9383:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode378899020
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 1866179473879517665
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis781610522
    kvnemesis.go:165: error applying x.AdminMerge(ctx, tk(7294806338321609923)) // kv/kvserver/replica_command.go:807: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.006s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]: kv/kvserver/replica_command.go:807: merge failed: waiting for all right-hand replicas to initialize: operation "wait for replicas init" timed out after 5.006s (given timeout 5s): grpc: context deadline exceeded [code 4/DeadlineExceeded]
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis781610522/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis781610522/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis781610522/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis781610522/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode378899020
--- FAIL: TestKVNemesisMultiNode (16.20s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv _{This test on roachdash | Improve this report!

Jira issue: CRDB-26424}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-04-05T07:07:01Z

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1 @ e847e4358c004d4a8977ae26749768ad1d417fba:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode408152875
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 1871784279202253796
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis124542662
    kvnemesis.go:165: error applying x.AdminSplit(ctx, tk(14849423503325755208)) // unable to find store 3 in range r62:/Table/{61-100/"01f7914b44c52225"} [(n4,s4):3, (n2,s2):2, next=6, gen=15]: unable to find store 3 in range r62:/Table/{61-100/"01f7914b44c52225"} [(n4,s4):3, (n2,s2):2, next=6, gen=15]
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis124542662/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis124542662/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis124542662/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis124542662/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode408152875
--- FAIL: TestKVNemesisMultiNode (15.37s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-05-01T11:39:58Z

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.1 @ fa8d237846052ae72cc44b3eaebe1f38c89834f8:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1348594126
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 949448558153492991
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis578809455
    kvnemesis.go:165: error applying x.ReverseScanForUpdate(ctx, tk(10883489528849899722), tk(13905202809364222425), 0) // WriteTooOldError: write for key /Table/100/"bc3c83ecf9bcabb2" at timestamp 1682941189.717989694,0 too old; wrote at 1682941189.891155495,2: {<nil> 0 {0xc009043c50} <nil> 1682941189.932605013,0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis578809455/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis578809455/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis578809455/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis578809455/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1348594126
--- FAIL: TestKVNemesisMultiNode (17.82s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

kv/kvnemesis: TestKVNemesisMultiNode failed #102411 kv/kvnemesis: TestKVNemesisMultiNode failed [C-test-failure O-robot T-kv branch-release-23.1.0]
kv/kvnemesis: TestKVNemesisMultiNode failed #101721 kv/kvnemesis: TestKVNemesisMultiNode failed [C-bug C-test-failure O-robot T-kv-replication branch-master]
kv/kvnemesis: TestKVNemesisMultiNode failed #101048 kv/kvnemesis: TestKVNemesisMultiNode failed [C-test-failure O-robot T-kv branch-release-22.2]

_{This test on roachdash | Improve this report!}

aliher1911 · 2023-05-02T11:29:22Z

Those errors above are different, so more than a single bug or flake, but overall problem is similar to what is happening on other branches mentioned.

github-actions · 2023-06-05T10:05:57Z

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 2, 2023

cockroach-teamcity added this to the 23.1 milestone Apr 2, 2023

blathers-crl bot added the T-kv KV Team label Apr 2, 2023

cockroach-teamcity mentioned this issue Apr 10, 2023

kv/kvnemesis: TestKVNemesisMultiNode failed #101048

Closed

cockroach-teamcity mentioned this issue Apr 18, 2023

kv/kvnemesis: TestKVNemesisMultiNode failed #101721

Closed

cockroach-teamcity mentioned this issue Apr 27, 2023

kv/kvnemesis: TestKVNemesisMultiNode failed #102411

Closed

aliher1911 added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label May 3, 2023

This was referenced May 28, 2023

kv/kvnemesis: TestKVNemesisMultiNode failed #104012

Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #104183

Closed

github-actions bot added the no-test-failure-activity label Jun 5, 2023

nvanbenschoten closed this as completed Jun 6, 2023

shralex added the X-stale label Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv/kvnemesis: TestKVNemesisMultiNode failed #100417

kv/kvnemesis: TestKVNemesisMultiNode failed #100417

cockroach-teamcity commented Apr 2, 2023 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Apr 5, 2023

cockroach-teamcity commented May 1, 2023

aliher1911 commented May 2, 2023

github-actions bot commented Jun 5, 2023

kv/kvnemesis: TestKVNemesisMultiNode failed #100417

kv/kvnemesis: TestKVNemesisMultiNode failed #100417

Comments

cockroach-teamcity commented Apr 2, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Apr 5, 2023

cockroach-teamcity commented May 1, 2023

aliher1911 commented May 2, 2023

github-actions bot commented Jun 5, 2023

cockroach-teamcity commented Apr 2, 2023 •

edited by cockroach-jira-scripts

Loading