Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #128263

Closed
cockroach-teamcity opened this issue Aug 3, 2024 · 5 comments
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #128263

cockroach-teamcity opened this issue Aug 3, 2024 · 5 comments
Labels
branch-release-23.2.9-rc C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-stale
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 3, 2024 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2901353638
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:320: seed: 1024139707414858014
    kvnemesis_test.go:227: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3453843062
    kvnemesis.go:177: range 114 (/Table/100/"807307fa36689451") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1722685912665696427 LockAge:0 GCBytesAge:1462 LiveBytes:63 LiveCount:1 KeyBytes:348 KeyCount:8 ValBytes:246 ValCount:15 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:1907 SysCount:15 AbortSpanBytes:811}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1722685912665696427 LockAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-10 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:212: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3453843062/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3453843062/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3453843062/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3453843062/kvs-scan.txt
    kvnemesis_test.go:348: [0] proposed: 3696
    kvnemesis_test.go:350: [0] reproposed unchanged: 61
    kvnemesis_test.go:352: [0] reproposed with new LAI: 16
    kvnemesis_test.go:348: [1] proposed: 475
    kvnemesis_test.go:350: [1] reproposed unchanged: 3
    kvnemesis_test.go:352: [1] reproposed with new LAI: 8
    kvnemesis_test.go:348: [2] proposed: 529
    kvnemesis_test.go:350: [2] reproposed unchanged: 7
    kvnemesis_test.go:352: [2] reproposed with new LAI: 9
    kvnemesis_test.go:348: [3] proposed: 162
    kvnemesis_test.go:350: [3] reproposed unchanged: 1
    kvnemesis_test.go:352: [3] reproposed with new LAI: 2
    kvnemesis_test.go:357: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:357
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:297
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2901353638
--- FAIL: TestKVNemesisMultiNode (28.55s)

Parameters:

  • TAGS=bazel,gss
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-40890
@cockroach-teamcity cockroach-teamcity added branch-release-23.2.9-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Aug 3, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 3, 2024
@kvoli
Copy link
Collaborator

kvoli commented Aug 5, 2024

This is identical to #124959 (comment)

I'm going to remove the release blocker, will triage further later.

@kvoli kvoli removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 5, 2024
@pav-kv
Copy link
Collaborator

pav-kv commented Aug 8, 2024

There is a long history with SysBytes:10 discrepancy, I wonder if the SysBytes:-10 here is similar/related.

#118471 (comment)
#93896

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 10, 2024
Closed
@kvoli
Copy link
Collaborator

kvoli commented Aug 12, 2024

Duplicate of #118471

@kvoli kvoli marked this as a duplicate of #118471 Aug 12, 2024
@kvoli kvoli added X-duplicate Closed as a duplicate of another issue. P-3 Issues/test failures with no fix SLA labels Aug 12, 2024
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

Fatal error:

panic: r73/1: closed timestamp regression 1723472806.515027238,0 -> 1723472805.922626233,0 (cmd "4adfcf3689f9ada4" index 14 term 6 (LAI=2 CTS=1723472805.922626233,0))
data: 004adfcf3689f9ada430016a13420092010e09e2cac06b6301eb1760a706680372760a74000000000000000002000000172f016b12ec123337346635623064383534613061613200ff01000174786e2d11760876acae4aad8e71b767f26da25a0002c901172f016b12ec123666616362643439666632316466353000ff01000174786e2de64b6f8e319d4147b6d40287ce08c7650002cb017a0020028a010a08b9b5c1c8abacc0f517 [recovered]
	panic: r73/1: closed timestamp regression 1723472806.515027238,0 -> 1723472805.922626233,0 (cmd "4adfcf3689f9ada4" index 14 term 6 (LAI=2 CTS=1723472805.922626233,0))
data: 004adfcf3689f9ada430016a13420092010e09e2cac06b6301eb1760a706680372760a74000000000000000002000000172f016b12ec123337346635623064383534613061613200ff01000174786e2d11760876acae4aad8e71b767f26da25a0002c901172f016b12ec123666616362643439666632316466353000ff01000174786e2de64b6f8e319d4147b6d40287ce08c7650002cb017a0020028a010a08b9b5c1c8abacc0f517

Stack:

goroutine 5065983 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f05b05?, {0x6c74e40, 0xc00cafeba0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x470c1e0?, 0xc00bc7f150?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply.func1({0x509ede6?, 0xc014d1d2c0?}, {0xc0061679c0?, 0x8?, 0xc0061678d0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:250 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply(0xc00a89eeb0, 0x1, {0xc006ebbb90, 0x8}, {0x6, 0xe, 0x0, {0xc00d3bafc0, 0xa9, 0xba}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:301 +0x5f5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Asserter).Apply(0x4f72140?, 0x41aa3d?, 0x490f240?, {0xc006ebbb90, 0x8}, {0x6, 0xe, 0x0, {0xc00d3bafc0, 0xa9, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:231 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.kvnemesisTestCfg.testClusterArgs.func7({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:146 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaApplyTestingFilters({0xc008085a10?, 0x6c74e40?}, 0xc01116b8f0?, 0xc00273c128?, {0xc008085a60?, 0xc012f44c30?, 0x0?}, 0x40?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:117 +0x1f7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc008085a10, {0x6c74e40, 0xc01116b8f0}, {0x6c94e20?, 0xc00273c128})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:113 +0xe9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter({0x6ca4110?, 0xc008085db8}, 0xc006168568)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc006168ad8, {0x6c74e40, 0xc00ad925a0}, {0x6ca4110, 0xc008085d88})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc006168ad8, {0x6c74e40, 0xc00ad925a0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1065 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:750 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc013b66000, 0x6c74e40?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc01378d8f0, {0x6c74e40, 0xc00cafeba0}, {0x6c58fc0, 0xc013b66000}, 0xc014f9d800)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6c74e40?, 0xc00cafeba0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 5035220
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode959338479
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:320: seed: 6214398497690885111
    kvnemesis_test.go:227: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4071051694

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 18, 2024
Closed
@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
@github-actions GitHub Actions
Copy link

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.2.9-rc C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-stale
Projects
No open projects
KV
Status: roachtest/unit test backlog
Milestone
23.2
Development

No branches or pull requests
3 participants
@pav-kv @cockroach-teamcity @kvoli