Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisSingleNode failed #117056

Closed
cockroach-teamcity opened this issue Dec 24, 2023 · 7 comments
Closed

kv/kvnemesis: TestKVNemesisSingleNode failed #117056

cockroach-teamcity opened this issue Dec 24, 2023 · 7 comments
Assignees
@arulajmani
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 24, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1328
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1462
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	Wraps: (2) panic applying step 
        	            	  | db0.DelRange(ctx, tk(1817280160006953746), tk(12042416779941932751), true /* @s60 */): lock table num=3
        	            	  |  lock: /Table/100/"50c26bd650bf7253"
        	            	  |    queued locking requests:
        	            	  |     active: false req: 85, strength: Shared, txn: f65a8727-ebdc-495a-bbd5-3c0b68b6020e
        	            	  |  lock: /Table/100/"53fe1efd360aa871"
        	            	  |   holder: txn: 354c15f5-4f14-45ef-9e5b-0ec8732d275c epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 84, strength: Shared, txn: none
        	            	  |    distinguished req: 84
        	            	  |  lock: /Table/100/"73fa57e6b1b4a094"
        	            	  |   holder: txn: 90deef1b-ad8c-496f-865c-f609cdf0e150 epoch: 0, iso: Snapshot, ts: 1703417218.761473730,0, info: unrepl [(str: Exclusive seq: 0)]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 85, strength: Shared, txn: f65a8727-ebdc-495a-bbd5-3c0b68b6020e
        	            	  |    distinguished req: 85
        	            	  |
        	            	  | error: queued locking request 84 does not conflict with holder/waiting requests  lock: /Table/100/"53fe1efd360aa871"
        	            	  |   holder: txn: 354c15f5-4f14-45ef-9e5b-0ec8732d275c epoch: 0, iso: Serializable, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 84, strength: Shared, txn: none
        	            	  |    distinguished req: 84
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode3812132112
--- FAIL: TestKVNemesisSingleNode (20.75s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-34942
@cockroach-teamcity cockroach-teamcity added branch-release-23.2.0-rc (deleted) C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Dec 24, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Dec 24, 2023
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

Fatal error:

panic: lock table num=11
 lock: /Table/100/"4db01cce83ab4150"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4de1dd0b92828c68"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4de919506670a56a"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4deba6bc25001af4"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4eafc4fb6c663a90"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"5856a9d545ce3189"
  empty
 lock: /Table/100/"595ae1383eb474af"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5b1cf5473897a111"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5fe303e75178d421"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"63db4aee478bb7b0"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"6a45accb56a9e08d"
  holder: txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 89, strength: Shared, txn: none
   distinguished req: 89

error: queued locking request 89 does not conflict with holder/waiting requests  lock: /Table/100/"6a45accb56a9e08d"
  holder: txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 89, strength: Shared, txn: none
   distinguished req: 89 [recovered]
	panic: lock table num=11
 lock: /Table/100/"4db01cce83ab4150"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4de1dd0b92828c68"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4de919506670a56a"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4deba6bc25001af4"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"4eafc4fb6c663a90"
   queued locking requests:
    active: false req: 85, strength: Intent, txn: 79a7f9a4-5123-4ee6-9704-76aec6ba70cd
 lock: /Table/100/"5856a9d545ce3189"
  empty
 lock: /Table/100/"595ae1383eb474af"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5b1cf5473897a111"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5fe303e75178d421"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"63db4aee478bb7b0"
  holders: txn: 1030a93c-4e73-433b-b656-3197adc85fda epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
           txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"6a45accb56a9e08d"
  holder: txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 89, strength: Shared, txn: none
   distinguished req: 89

error: queued locking request 89 does not conflict with holder/waiting requests  lock: /Table/100/"6a45accb56a9e08d"
  holder: txn: 01b65c7b-83c4-43d7-a222-7d1d2b6b1c94 epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 89, strength: Shared, txn: none
   distinguished req: 89

Stack:

goroutine 915041 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1ee2da5?, {0x6b69860, 0xc00d38c150})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46a5c40?, 0xc00638b0d0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc002a29400)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7fc45fe62480?, 0xc002a29400?}}, 0x2?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x7fc4aa86c108?, {0x6b69860, 0xc0051f9ef0}, 0xd3eda5?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc002d42c80, {0x6b69860, 0xc0051f9ef0}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x884
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc002d42d88, {0x6b69860, 0xc0051f9ef0}, {0x6ba8998?, 0xc001dfea08})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x828
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fc45fe62810?, 0xc002d43168}, 0xc00604a560)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc00604aae8, {0x6b69860, 0xc00b80da10}, {0x6b98a30, 0xc002d43108})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc00604aae8, {0x6b69860, 0xc00b80da10})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc00353e000, 0xc0114d5140?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc010773dc0, {0x6b69860, 0xc00d38c150}, {0x6b4da80, 0xc00353e000}, 0xc002533500)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6b69860?, 0xc00d38c150?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 902511
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode1604848176
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 6716102886854031454
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3935783961

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

Fatal error:

panic: lock table num=4
 lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163
 lock: /Table/100/"4265b493b0e2a846"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"4c9c8293c3c611a2"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"57a3beb235aa78ea"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]

error: queued locking request 163 does not conflict with holder/waiting requests  lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163 [recovered]
	panic: lock table num=4
 lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163
 lock: /Table/100/"4265b493b0e2a846"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"4c9c8293c3c611a2"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"57a3beb235aa78ea"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]

error: queued locking request 163 does not conflict with holder/waiting requests  lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163 [recovered]
	panic: lock table num=4
 lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163
 lock: /Table/100/"4265b493b0e2a846"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"4c9c8293c3c611a2"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"57a3beb235aa78ea"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]

error: queued locking request 163 does not conflict with holder/waiting requests  lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163 [recovered]
	panic: lock table num=4
 lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163
 lock: /Table/100/"4265b493b0e2a846"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"4c9c8293c3c611a2"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
 lock: /Table/100/"57a3beb235aa78ea"
  holder: txn: 67549aea-b0e3-45f7-bc05-7a80e55e6120 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]

error: queued locking request 163 does not conflict with holder/waiting requests  lock: /Table/100/"38308838b18c1042"
  holder: txn: 175221f2-cf55-4881-afe4-347e6c37ec46 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 163, strength: Shared, txn: c66398dd-9262-4641-a047-f19869cf3c7b
   distinguished req: 163

Stack:

goroutine 6210 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1b04905?, {0x6b62740, 0xc0050a60f0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x469d360?, 0xc005b1a700?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc004320000?, {0x6b62740, 0xc0050a6600})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x469d360?, 0xc005b1a700?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x14e
panic({0x469d360?, 0xc005b1a700?})
	GOROOT/src/runtime/panic.go:920 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0027d6c80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x1d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7f68142862d8?, 0xc0027d6c80?}}, 0xc0034843c0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x1?, {0x6b62740, 0xc0050a68d0}, 0x5e?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0027db900, {0x6b62740, 0xc0050a68d0}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x884
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc0027db900, {0x6b62740?, 0xc0050a68d0}, 0xc0025cc240, 0xc00638aa80, 0xc003485850, {{0x17a54a919aced84a, 0x0, 0x0}, {0x17a54a919a3641ca, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:153 +0x1790
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc0027db900, {0x6b62740, 0xc0050a68d0}, 0xc0025cc240, 0xc00638aa80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:179 +0x685
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0027db900, {0x6b62740, 0xc0050a68d0}, 0xc0025cc240, 0x52dc1f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:511 +0x3a3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0027db900, {0x6b62740?, 0xc0050a6780?}, 0xc0025cc240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x725
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc002560a80, {0x6b62740?, 0xc0050a66f0?}, 0xc0025cc240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x6b62740, 0xc0050a66f0}, 0xc0025cc240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001cd6400, {0x6b62740?, 0xc0050a6660?}, {0xc00625bc20?}, 0xc0025cc240)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x545
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001cd6400, {0x6b62740, 0xc0050a6600}, 0xc0025cc240)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1462 +0x2b5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x45
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x6b62740, 0xc0050a6600}, {0x4f29f00, 0xc0025cc240}, 0xc003bfd340, 0xc002b6a6d8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x6b62740, 0xc0050a6600}, {0x4f29f00, 0xc0025cc240}, 0xc0050a6600?, 0xc003bfd360)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc004320000?, {{0x491dd8?}, {0x6b99730?, 0xc001c778f0?}}}, {0x6b62740, 0xc0050a6600}, {0x4f29f00, 0xc0025cc240}, 0xc003bfd340, 0xc003bfd380)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x6b62740?, 0xc0050a6600?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003ba34d0, {0x6b62740?, 0xc0050a6600}, {0x0?, 0x1?}, 0xc00779fc80)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?}, 0x5583a5?, 0x7f68143d2800?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b62740?, 0xc0050a6600?}, {0x4f29f00?, 0xc0025cc240?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x6b62740?, 0xc0050a6600?}, {0xc003e2d1e0?, 0xc00779fe08?}, {0x4f29f00?, 0xc0025cc240?}, {0x4e05c00?, 0xc004be4180}, 0xc00121a9b0?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x51
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0x6b62740, 0xc0050a6600}, {0x5010e40, 0x21}, {0x4f29f00, 0xc0025cc240}, {0x4e05c00, 0xc004be4180}, 0x14ab9f2?, 0xc005b1bac0, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x47a
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0x6b62740, 0xc0050a6600}, {0x5010e40, 0x21}, {0x4f29f00, 0xc0025cc240}, {0x4e05c00, 0xc004be4180}, 0x24?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x14e
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0x6b62740, 0xc0050a6180}, 0xc0025cc120, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x32f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00535cfc0, {0x6b62740, 0xc0050a6180}, 0x31?, {0x6b47260, 0xc003097800?}, 0xc0025cc120)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x1bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00535cfc0, {0x6b62740, 0xc0050a6180}, 0x6b62740?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x87
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0025ac100, {0x6b62740, 0xc0050a6180}, 0xc0025cc000?, {0xc0056d73b0, 0xc005ecbc70, 0xc005ecbce0, 0x0, 0x0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2419 +0x11f6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0025ac100, {0x6b62740?, 0xc0050a6180}, 0xc0025cc000, {{0xc0060c7620, 0x14, 0x18}, {0xc0060c7638, 0x14, 0x18}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1920 +0x7b2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0025ac100, {0x6b62740?, 0xc0050a6180}, 0xc0025cc000, {{0xc0060c7620, 0x14, 0x18}, {0xc0060c7638, 0x14, 0x18}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1488 +0x3d0
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0025ac100, {0x6b627b0, 0xc0037cc070}, 0xc0025cc000)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1104 +0x658
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc0060dca68, {0x6b627b0, 0xc0037cc070}, 0xc0025cc000)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:224 +0xa2
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).sendBatch.func1.1({0x6b627b0, 0xc0037cc070})
	github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:334 +0xa2
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x6b62740?, 0xc0050a60f0?}, {0xc00539ea80, 0x2a}, 0xdf8475800, 0xc0048e4d80)
	github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0xc9
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).sendBatch.func1.2({0x6b62740, 0xc0050a60f0})
	github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:352 +0xea
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).sendBatch.func1({0x6b62740, 0xc0050a60f0})
	github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:374 +0x226
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 2978
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode1739999097
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 6587930031061839152
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3707153198

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 8496f8fa0f076f739862b6e9ceace125a8eb2d93:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	Wraps: (2) panic applying step 
        	            	  | db0.DelRangeUsingTombstone(ctx, tk(6090837080529301569), tk(14548274780175895393) /* @s38 */): lock table num=13
        	            	  |  lock: /Table/100/"5b27eb9890142b33"
        	            	  |    queued locking requests:
        	            	  |     active: false req: 37, strength: Shared, txn: 9858635f-7dff-4fa9-8ba1-b18bfaca55c6
        	            	  |     active: true req: 42, strength: Intent, txn: 2fb9cf98-ac2c-4649-bc65-443b61037f2b
        	            	  |    distinguished req: 42
        	            	  |  lock: /Table/100/"5cda283662d57dca"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 37, strength: Shared, txn: 9858635f-7dff-4fa9-8ba1-b18bfaca55c6
        	            	  |    distinguished req: 37
        	            	  |  lock: /Table/100/"5ea5517e57c8348a"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"6405b89793623b3b"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"6d508680e0dc2d57"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"71db8f56bf67d3ed"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"7547299306e36b11"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"85456bdb3cee0b3f"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"9a9a02e158f3f09f"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"a4eac0ac1bc43e63"
        	            	  |   holders: txn: db880509-43b6-45d9-a08f-ba94baa89d9f epoch: 0, iso: Serializable, info: repl [Shared]
        	            	  |            txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"a734ca8d1b48f326"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"ab6e6eef4f769748"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"c8857f766aa96c91"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
        	            	  |
        	            	  | error: queued locking request 37 does not conflict with holder/waiting requests  lock: /Table/100/"5cda283662d57dca"
        	            	  |   holder: txn: 23302027-012c-48e1-94ec-08952f55509e epoch: 0, iso: Serializable, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 37, strength: Shared, txn: 9858635f-7dff-4fa9-8ba1-b18bfaca55c6
        	            	  |    distinguished req: 37
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2488919947
--- FAIL: TestKVNemesisSingleNode (4.35s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 8496f8fa0f076f739862b6e9ceace125a8eb2d93:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
        	            	Wraps: (2) panic applying step 
        	            	  | db0.ScanForUpdate(ctx, tk(8060142350040730467), tk(15611365963574080292), 0): lock table num=5
        	            	  |  lock: /Table/100/"7fba1f45b4f664f9"
        	            	  |    queued locking requests:
        	            	  |     active: false req: 59, strength: Shared, txn: f63cdff9-aad8-4ca8-8710-a6748c818108
        	            	  |     active: true req: 69, strength: Intent, txn: 3bc6439f-ea1a-4c63-90b4-dcac5766e0d4
        	            	  |    distinguished req: 69
        	            	  |  lock: /Table/100/"d2417493c3d9f52c"
        	            	  |    queued locking requests:
        	            	  |     active: false req: 59, strength: Shared, txn: f63cdff9-aad8-4ca8-8710-a6748c818108
        	            	  |     active: true req: 61, strength: Exclusive, txn: da4744b2-d6b9-4030-8d62-f4ea2efeb9f1
        	            	  |    distinguished req: 61
        	            	  |  lock: /Table/100/"d990741abbdde5ca"
        	            	  |   holder: txn: f8b79d01-19aa-4456-8409-dd87ad266927 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 59, strength: Shared, txn: f63cdff9-aad8-4ca8-8710-a6748c818108
        	            	  |    distinguished req: 59
        	            	  |  lock: /Table/100/"db6a06c96fa1a445"
        	            	  |   holder: txn: f8b79d01-19aa-4456-8409-dd87ad266927 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"e0dfc7ec392a4aef"
        	            	  |   holder: txn: f8b79d01-19aa-4456-8409-dd87ad266927 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
        	            	  |
        	            	  | error: queued locking request 59 does not conflict with holder/waiting requests  lock: /Table/100/"d990741abbdde5ca"
        	            	  |   holder: txn: f8b79d01-19aa-4456-8409-dd87ad266927 epoch: 0, iso: Snapshot, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 59, strength: Shared, txn: f63cdff9-aad8-4ca8-8710-a6748c818108
        	            	  |    distinguished req: 59
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode1762249728
--- FAIL: TestKVNemesisSingleNode (49.13s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on release-23.2.0-rc @ 8496f8fa0f076f739862b6e9ceace125a8eb2d93:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1328
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1462
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	Wraps: (2) panic applying step 
        	            	  | db0.DelRange(ctx, tk(1587037523333936976), tk(4770206123986050133), true /* @s36 */): lock table num=3
        	            	  |  lock: /Table/100/"29f4d51f27a4ec05"
        	            	  |   holder: txn: bd3246d8-466a-4a5c-9ff5-f658d6bebd0f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 143, strength: Shared, txn: aad40dd2-5bf2-4816-bbd4-0f0a2cc05c31
        	            	  |    distinguished req: 143
        	            	  |  lock: /Table/100/"3b12d55f2b70c418"
        	            	  |   holder: txn: 7a2cdcf1-81a7-417e-ac21-037f2f2b8b3c epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
        	            	  |  lock: /Table/100/"418889852777352a"
        	            	  |   holder: txn: 7a2cdcf1-81a7-417e-ac21-037f2f2b8b3c epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]
        	            	  |
        	            	  | error: queued locking request 143 does not conflict with holder/waiting requests  lock: /Table/100/"29f4d51f27a4ec05"
        	            	  |   holder: txn: bd3246d8-466a-4a5c-9ff5-f658d6bebd0f epoch: 0, iso: ReadCommitted, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 143, strength: Shared, txn: aad40dd2-5bf2-4816-bbd4-0f0a2cc05c31
        	            	  |    distinguished req: 143
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode234076245
--- FAIL: TestKVNemesisSingleNode (9.18s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 4, 2024
@exalate-issue-sync exalate-issue-sync bot added the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 4, 2024
@arulajmani
Copy link
Collaborator

#116242 hadn't been backported to release-23.2.0; opened #117329

@arulajmani arulajmani removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. branch-release-23.2.0-rc (deleted) labels Jan 4, 2024
@nvanbenschoten nvanbenschoten added the X-duplicate Closed as a duplicate of another issue. label Jan 11, 2024
@nvanbenschoten
Copy link
Member

Closing as X-Duplicate; consolidating over at #115694.

@nvanbenschoten nvanbenschoten closed this as not planned Won't fix, can't repro, duplicate, stale Jan 11, 2024
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Projects
No open projects
KV
Status: Closed
Milestone
23.2
Development

No branches or pull requests
3 participants
@nvanbenschoten @arulajmani @cockroach-teamcity