kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

cockroach-teamcity · 2023-12-05T13:00:01Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ df4b4e856c33477cdad4a659adba959987c488ca:

Fatal error:

panic: lock table num=2
 lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299
 lock: /Table/100/"f91ee672f2abf20d"
  holder: txn: 11525b9b-4822-4097-91a1-486697cd79e1 epoch: 0, iso: ReadCommitted, ts: 1701781195.664329888,1, info: repl [Intent]
   queued locking requests:
    active: true req: 254, strength: Intent, txn: none
   distinguished req: 254

error: queued locking request 299 does not conflict with holder/waiting requests  lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299 [recovered]
	panic: lock table num=2
 lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299
 lock: /Table/100/"f91ee672f2abf20d"
  holder: txn: 11525b9b-4822-4097-91a1-486697cd79e1 epoch: 0, iso: ReadCommitted, ts: 1701781195.664329888,1, info: repl [Intent]
   queued locking requests:
    active: true req: 254, strength: Intent, txn: none
   distinguished req: 254

error: queued locking request 299 does not conflict with holder/waiting requests  lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299

Stack:

goroutine 3112257 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0122c76e0?, {0x6d9be30, 0xc013d5ec00})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48efba0, 0xc00f27c2e0})
	GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0003f0280)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7fda51a7bca0?, 0xc0003f0280?}}, 0x1?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x94
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x52439d8?, {0x6d9be30, 0xc00fa75cb0}, 0x6?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x36
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc002806580, {0x6d9be30, 0xc00fa75cb0}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x802
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc002806688, {0x6d9be30, 0xc00fa75cb0}, {0x6ddaac8?, 0xc004054008})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x848
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fda4f599c20?, 0xc002806a68}, 0xc0122c8570)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0122c8ae8, {0x6d9be30, 0xc0103c85a0}, {0x6dcae20, 0xc002806a08})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1f8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0122c8ae8, {0x6d9be30, 0xc0103c85a0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc006824000, 0xc0170d71a0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc000fd9f10, {0x6d9be30, 0xc013d5ec00}, {0x6d81820, 0xc006824000}, 0xc00f261500)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6d9be30?, 0xc013d5ec00?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b

Log preceding fatal error

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos734143989
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 750006408177795411
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis429571242

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv _{This test on roachdash | Improve this report!

Jira issue: CRDB-34118}

The text was updated successfully, but these errors were encountered:

erikgrinaker · 2023-12-05T13:03:37Z

@arulajmani Related to the recent lock table assertions you've added?

cockroach-teamcity · 2023-12-05T13:22:04Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ df4b4e856c33477cdad4a659adba959987c488ca:

        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
        	            	  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
        	            	Wraps: (2) panic applying step 
        	            	  | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
        	            	  |   txn.SetIsoLevel(isolation.ReadCommitted)
        	            	  |   txn.DelRange(ctx, tk(8543534448051736489), tk(9408248879097924965), true /* @s114 */) // @1701782463.701752092,0 <nil>
        	            	  |   txn.ReverseScanForUpdateGuaranteedDurability(ctx, tk(10993648730429643591), tk(18161759648309122977), 0) // @1701782463.703782197,0 (/Table/100/"f9257238331e0830":v34, /Table/100/"ed3e32e24bc0dc5c":v34, <nil>)
        	            	  |   txn.ReverseScanForShareGuaranteedDurability(ctx, tk(8543631286166663424), tk(17421013793695485217), 0)
        	            	  |   txn.Get(ctx, tk(17762943607935793602))
        	            	  |   return errors.New("rollback")
        	            	  | }): lock table num=4
        	            	  |  lock: /Table/100/"3c2b77024c8063fd"
        	            	  |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 457, strength: Shared, txn: none
        	            	  |    distinguished req: 457
        	            	  |  lock: /Table/100/"47da129319a8446b"
        	            	  |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"5967f24f663056f2"
        	            	  |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |  lock: /Table/100/"5d031b446cfd8fa1"
        	            	  |   holders: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |            txn: 1a6e8dba-16a0-42ca-9196-9f94736f2cb2 epoch: 0, iso: ReadCommitted, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 453, strength: Intent, txn: none
        	            	  |    distinguished req: 453
        	            	  |
        	            	  | error: queued locking request 457 does not conflict with holder/waiting requests  lock: /Table/100/"3c2b77024c8063fd"
        	            	  |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 457, strength: Shared, txn: none
        	            	  |    distinguished req: 457
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos1839409175
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (44.22s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

arulajmani · 2023-12-05T14:03:48Z

Yeah, this looks related to the verification I added. Looking into this, but I'll re-label this as a GA-blocker given we're only seeing this with shared locks.

arulajmani · 2023-12-05T16:55:20Z

I have a plausible explanation for what might be going on here. In both these cases, we see the panic is coming from a lock held by a finalized transaction. For example,

        	            	  |  lock: /Table/100/"3c2b77024c8063fd"
        	            	  |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 457, strength: Shared, txn: none
        	            	  |    distinguished req: 457

Note that the shared lock held by txn b59e7e22 is replicated. That means it must have been pulled into the lock table by a conflicting request. My guess is that this must be a non-transactional[1] exclusive/intent locking request.

That would explain why req 457 is waiting on this key. Once txn b59e7e22 is finalized, the non-transactional exclusive/intent locking request returns from its push and resumes its scan. As it no longer conflicts with the lock, and its a non-transactional request, its removed from the lock's wait queue. This would then explain the lock state we see the test fail with.

This isn't really a correctness issue, even though we're tripping verification here. To that end, I'll remove the GA-blocker label. I'm not sure what the best fix should be here yet.

[1] If it were a transactional request, we'd expect it to still be in the lock's wait queue as an inactive waiter.

cockroach-teamcity · 2023-12-07T14:39:38Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 75dbd5b7dd05e885fd729e3bb13622c8f8b74b85:

Fatal error:

panic: lock table num=6
 lock: /Table/100/"cc4dd6d011c63d0b"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"cd1ed982d875f840"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62
 lock: /Table/100/"e55f9f6ba0074cee"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7d992cf5d406ad2"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e8486f53f08cba7a"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 62 does not conflict with holder/waiting requests  lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62 [recovered]
	panic: lock table num=6
 lock: /Table/100/"cc4dd6d011c63d0b"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"cd1ed982d875f840"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62
 lock: /Table/100/"e55f9f6ba0074cee"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7d992cf5d406ad2"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e8486f53f08cba7a"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 62 does not conflict with holder/waiting requests  lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62

Stack:

goroutine 3177922 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x200b86a?, {0x6d9def0, 0xc00862bc20})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48f14a0, 0xc00bb90d60})
	GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc000800f00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7f1e824b2450?, 0xc000800f00?}}, 0x1?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x94
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x52455f1?, {0x6d9def0, 0xc001144960}, 0x6?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x36
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0098c2c80, {0x6d9def0, 0xc001144960}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x802
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc0098c2d88, {0x6d9def0, 0xc001144960}, {0x6ddcb88?, 0xc002924508})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x848
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f1e83252740?, 0xc0098c3168}, 0xc003fec570)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc003fecae8, {0x6d9def0, 0xc015f66990}, {0x6dccee0, 0xc0098c3108})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1f8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc003fecae8, {0x6d9def0, 0xc015f66990})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0059f0a80, 0xc012256ae0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00d6b75e0, {0x6d9def0, 0xc00862bc20}, {0x6d838e0, 0xc0059f0a80}, 0xc00825b500)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6d9def0?, 0xc00862bc20?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b

Log preceding fatal error

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos125762160
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 29713711782940073
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1233562520

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-12-11T11:08:42Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 43a2532550ca7abfc3c5682df31b0cc20da934b0:

        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
        	            	  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
        	            	Wraps: (2) panic applying step 
        	            	  | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
        	            	  |   txn.SetIsoLevel(isolation.Snapshot)
        	            	  |   txn.ScanForShare(ctx, tk(10161750788992158904), tk(10607603416092531769), 0) // @1702292838.076731702,0 <nil>
        	            	  |   txn.ReverseScanForShareSkipLocked(ctx, tk(12325901490205422963), tk(16583471604304201683), 0) // @1702292838.076731702,0 (/Table/100/"e4a079ee176e85ad":v85, /Table/100/"e31eb55758d3c35a":v89, <nil>)
        	            	  |   txn.Get(ctx, tk(3403608216299324618)) // @1702292838.076731702,0 (<nil>, <nil>)
        	            	  |   txn.GetForUpdate(ctx, tk(17532841644207285003)) // @1702292838.076731702,0 (<nil>, <nil>)
        	            	  |   txn.ReverseScanForUpdate(ctx, tk(6132161645053430098), tk(7977877486819318195), 0) // @1702292838.076731702,0 (/Table/100/"6a00e953313860af":v72, <nil>)
        	            	  |   b := &kv.Batch{}
        	            	  |   b.ReverseScanForShareGuaranteedDurability(tk(6560877028758136323), tk(12124959853369502821))
        	            	  |   b.ReverseScanForUpdateGuaranteedDurability(tk(6822100593481620689), tk(8187737862183061801))
        	            	  |   txn.CommitInBatch(ctx, b)
        	            	  |   return nil
        	            	  | }): lock table num=4
        	            	  |  lock: /Table/100/"41a761029fa3b064"
        	            	  |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
        	            	  |  lock: /Table/100/"4963dbe46f3295b0"
        	            	  |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
        	            	  |  lock: /Table/100/"6a00e953313860af"
        	            	  |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 152, strength: Shared, txn: e9314727-f793-4592-a65e-fbf6804c6837
        	            	  |    distinguished req: 152
        	            	  |  lock: /Table/100/"7161200d7eff3df8"
        	            	  |    queued locking requests:
        	            	  |     active: false req: 152, strength: Exclusive, txn: e9314727-f793-4592-a65e-fbf6804c6837
        	            	  |
        	            	  | error: queued locking request 152 does not conflict with holder/waiting requests  lock: /Table/100/"6a00e953313860af"
        	            	  |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 152, strength: Shared, txn: e9314727-f793-4592-a65e-fbf6804c6837
        	            	  |    distinguished req: 152
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos2936116515
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (45.92s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-12-12T13:56:03Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ caaa842c957d8487be32504cdd6379745f1a3ff6:

Fatal error:

panic: lock table num=20
 lock: /Table/100/"b2d30b8a0853de46"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
    active: true req: 25, strength: Exclusive, txn: 0f10bc94-1797-4806-ae47-2b8fbc358cf3
   distinguished req: 25
 lock: /Table/100/"bef75b44f626448c"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
 lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21
 lock: /Table/100/"e0c6ae0ae7889ef0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e0db12f1599718f8"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e14682aeacfaa0da"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1a38d86c350b2cf"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1bcdaf993d23b26"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e2e1ff711fa566ed"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e3bc17591f7941fe"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e4a630f18f44509b"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e62ce91b7825c690"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e6622f653b6dfce0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7b8227548d1a1b1"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e86983ea7356c4d5"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e910e5cf4cd49bd3"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eab81f7152d0ec32"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee6e807e54c65f1f"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee855cac35ddba9e"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eea34826ecb3ddc0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 21 does not conflict with holder/waiting requests  lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21 [recovered]
	panic: lock table num=20
 lock: /Table/100/"b2d30b8a0853de46"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
    active: true req: 25, strength: Exclusive, txn: 0f10bc94-1797-4806-ae47-2b8fbc358cf3
   distinguished req: 25
 lock: /Table/100/"bef75b44f626448c"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
 lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21
 lock: /Table/100/"e0c6ae0ae7889ef0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e0db12f1599718f8"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e14682aeacfaa0da"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1a38d86c350b2cf"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1bcdaf993d23b26"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e2e1ff711fa566ed"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e3bc17591f7941fe"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e4a630f18f44509b"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e62ce91b7825c690"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e6622f653b6dfce0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7b8227548d1a1b1"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e86983ea7356c4d5"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e910e5cf4cd49bd3"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eab81f7152d0ec32"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee6e807e54c65f1f"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee855cac35ddba9e"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eea34826ecb3ddc0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 21 does not conflict with holder/waiting requests  lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21

Stack:

goroutine 2201668 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x200ea2a?, {0x6da62f0, 0xc00ac74960})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48f79c0, 0xc00c903970})
	GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0105a2dc0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.Dequeue({{0x7f16183d2088?, 0xc0105a2dc0?}}, {0x6dd5418?, 0xc004112280?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:77 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).FinishReq(0xc005a00640?, 0xc0009f5bc0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:453 +0x106
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done(0xc00b4bb960, {0x6da62f0?, 0xc008305350?}, 0x0?, 0x0?, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1367 +0x143
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ProposalData).finishApplication(0xc00b4bb900, {0x6da62f0?, 0xc008305350?}, {0xc01436a880, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:242 +0x53
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicatedCmd).AckOutcomeAndFinish(0xc001ed6508, {0x6da62f0?, 0xc008305350?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_cmd.go:156 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.forEachAppliedCmdIter({0x6da62f0, 0xc008305350}, {0x7f1617b6edd0?, 0xc0093096b8?}, 0x55033b8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:268 +0xea
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc00076cae8, {0x6da62f0, 0xc008305350}, {0x6dd5340, 0xc009309688})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:301 +0x255
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc00076cae8, {0x6da62f0, 0xc008305350})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0041cb500, 0xc0110fc090?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0010eb3b0, {0x6da62f0, 0xc00ac74960}, {0x6d8bce0, 0xc0041cb500}, 0xc00fcfc000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6da62f0?, 0xc00ac74960?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b

Log preceding fatal error

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos1525528943
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 6622495634950520862
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3805446582

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

We've seen KVNemesis fail regularly because of a specific validation error. For now, quieten the assertion until the issue is fixed -- this ensures legit bugs aren't drowned out by the noise. Informs cockroachdb#115694 Informs cockroachdb#115598 Release note: None

116090: serverccl: skip data distribution test under race r=maryliag a=dhartunian Resolves: #115932 Epic: None Release note: None 116125: sql: skip a couple of tests under deadlock r=yuzefovich a=yuzefovich `TestExplainRedact` and `TestExplainRedactDDL` are quite long, making that some KV level deadlock mechanism fires incorrectly. We've skipped other tests due to this, so this commit skips these two as well. Fixes: #115861. Fixes: #115998. Release note: None 116242: concurrency: quieten KVNemesis failure because of validation r=arulajmani a=arulajmani We've seen KVNemesis fail regularly because of a specific validation error. For now, quieten the assertion until the issue is fixed -- this ensures legit bugs aren't drowned out by the noise. Informs #115694 Informs #115598 Release note: None Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Yahor Yuzefovich <[email protected]> Co-authored-by: Arul Ajmani <[email protected]>

We've seen KVNemesis fail regularly because of a specific validation error. For now, quieten the assertion until the issue is fixed -- this ensures legit bugs aren't drowned out by the noise. Informs #115694 Informs #115598 Release note: None

cockroach-teamcity · 2023-12-13T14:41:18Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ bc665a398d2ddbd27b60119e8272f49d0abf78a2:

        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
        	            	Wraps: (2) panic applying step 
        	            	  | db0.ReverseScanForUpdate(ctx, tk(5234225380589050268), tk(7787489959389179047), 0): lock table num=4
        	            	  |  lock: /Table/100/"166554c6bbe0c7f2"
        	            	  |   holder: txn: decf2ce2-ea91-47fa-8813-298360ed21ab epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 370, strength: Shared, txn: none
        	            	  |    distinguished req: 370
        	            	  |  lock: /Table/100/"2b18e1a7ddcdec02"
        	            	  |   holder: txn: b242c9ba-a917-493d-a1e4-9bd2c5711ce7 epoch: 0, iso: Serializable, ts: 1702478437.087277341,0, info: repl [Intent]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 361, strength: Intent, txn: none
        	            	  |    distinguished req: 361
        	            	  |  lock: /Table/100/"9f4bff7f0c26631c"
        	            	  |   holder: txn: 6c4f93af-3ac0-4629-8a8b-ca20488b6523 epoch: 0, iso: ReadCommitted, info: unrepl [(str: Shared seq: 0)]
        	            	  |  lock: /Table/100/"c1ff33290a82bc24"
        	            	  |   holder: txn: 6c4f93af-3ac0-4629-8a8b-ca20488b6523 epoch: 0, iso: ReadCommitted, info: unrepl [(str: Shared seq: 0)]
        	            	  |
        	            	  | error: queued locking request 370 does not conflict with holder/waiting requests  lock: /Table/100/"166554c6bbe0c7f2"
        	            	  |   holder: txn: decf2ce2-ea91-47fa-8813-298360ed21ab epoch: 0, iso: ReadCommitted, info: repl [Shared]
        	            	  |    queued locking requests:
        	            	  |     active: true req: 370, strength: Shared, txn: none
        	            	  |    distinguished req: 370
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos4110185987
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (8.70s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #116322 kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed [C-test-failure O-robot T-kv branch-release-23.2.0-rc release-blocker]

_{This test on roachdash | Improve this report!}

We've seen KVNemesis fail regularly because of a specific validation error. For now, quieten the assertion until the issue is fixed -- this ensures legit bugs aren't drowned out by the noise. Informs #115694 Informs #115598 Release note: None

github-actions · 2024-03-04T10:34:07Z

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

cockroach-teamcity · 2024-03-08T13:16:42Z

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 84f0e5f4625f8a2155d531251e0df6db8a5ef053:

Fatal error:

panic: r80/1: closed timestamp regression 1709903797.620507717,0 -> 1709903797.618308551,0 (cmd "2d34d0a7edc23f62" index 15 term 6 (LAI=2 CTS=1709903797.618308551,0))
data: 002d34d0a7edc23f6230016a154200920110098cb1ce7672ccba17800102a8018602729e010a9b01000000000000000001000000012e017a6b12ec123565636338306539346531376635316600ff01000100019f7a0c7469324976b4e9d767de021fad125e0a460a109f7a0c7469324976b4e9d767de021fad1a14ec123165313132323636646233346335336400012a0a088ce3bab6a78eb3dd1730e1ce324a0a088ce3bab6a78eb3dd175001120c088ce3bab6a78eb3dd17100018002000280048017a008201260a12637264622d7472616365722d7370616e69641210313236613666343533313638363939638201270a13637264622d7472616365722d747261636569641210343837626665646230643136306536378201080a0372656312017620028a010a08c7eb85c99c8eb3dd17 [recovered]
	panic: r80/1: closed timestamp regression 1709903797.620507717,0 -> 1709903797.618308551,0 (cmd "2d34d0a7edc23f62" index 15 term 6 (LAI=2 CTS=1709903797.618308551,0))
data: 002d34d0a7edc23f6230016a154200920110098cb1ce7672ccba17800102a8018602729e010a9b01000000000000000001000000012e017a6b12ec123565636338306539346531376635316600ff01000100019f7a0c7469324976b4e9d767de021fad125e0a460a109f7a0c7469324976b4e9d767de021fad1a14ec123165313132323636646233346335336400012a0a088ce3bab6a78eb3dd1730e1ce324a0a088ce3bab6a78eb3dd175001120c088ce3bab6a78eb3dd17100018002000280048017a008201260a12637264622d7472616365722d7370616e69641210313236613666343533313638363939638201270a13637264622d7472616365722d747261636569641210343837626665646230643136306536378201080a0372656312017620028a010a08c7eb85c99c8eb3dd17

Stack:

goroutine 11038 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f14b45?, {0x6c21ae0, 0xc005e65050})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46d79c0?, 0xc004e30350?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply.func1({0x5065b6d?, 0xc005ea4ae0?}, {0xc00442f9f0?, 0x8?, 0xc00442f650?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:250 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply(0xc006c98c80, 0x1, {0xc0063647d8, 0x8}, {0x6, 0xf, 0x0, {0xc00973e160, 0x132, 0x143}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:301 +0x5ff
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Asserter).Apply(0x4fde1c?, 0x89122448912244?, 0x492001?, {0xc0063647d8, 0x8}, {0x6, 0xf, 0x0, {0xc00973e160, 0x132, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:231 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.kvnemesisTestCfg.testClusterArgs.func7({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:145 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaApplyTestingFilters({0xc001e2ad90?, 0x6c21ae0?}, 0xc004cd3bf0?, 0xc001a78248?, {0xc001e2ade0?, 0xc005754640?, 0x0?}, 0xe0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:117 +0x1f7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc001e2ad90, {0x6c21ae0, 0xc004cd3bf0}, {0x6c41a80?, 0xc001a78248})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:113 +0xe9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter({0x6c50cb0?, 0xc001e2b138}, 0xc004430598)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc004430af8, {0x6c21ae0, 0xc005d47f20}, {0x6c50cb0, 0xc001e2b108})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc004430af8, {0x6c21ae0, 0xc005d47f20})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1059 +0x13c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:744 +0x1b8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0029f8a80, 0xc0052aeae0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc003668540, {0x6c21ae0, 0xc005e65050}, {0x6c05c60, 0xc0029f8a80}, 0xc0029f8000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x1e6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6c21ae0?, 0xc005e65050?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10752
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415

Log preceding fatal error

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos2182480276
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:319: seed: 6437887958677442706
    kvnemesis_test.go:226: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2296613533

Parameters:

TAGS=bazel,gss
stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

github-actions · 2024-04-15T10:02:26Z

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

cockroach-teamcity added this to the 23.2 milestone Dec 5, 2023

erikgrinaker assigned arulajmani Dec 5, 2023

erikgrinaker added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-kv-transactions Relating to MVCC and the transactional model. labels Dec 5, 2023

arulajmani added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Dec 5, 2023

arulajmani removed the GA-blocker label Dec 5, 2023

arulajmani mentioned this issue Dec 6, 2023

kv/kvnemesis: TestKVNemesisSingleNode failed #115694

Closed

arulajmani mentioned this issue Dec 12, 2023

concurrency: quieten KVNemesis failure because of validation #116242

Merged

blathers-crl bot mentioned this issue Dec 12, 2023

release-23.2: concurrency: quieten KVNemesis failure because of validation #116249

Merged

cockroach-teamcity mentioned this issue Dec 13, 2023

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #116322

Closed

cockroach-teamcity mentioned this issue Dec 26, 2023

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #117087

Closed

nvanbenschoten added the P-1 Issues/test failures with a fix SLA of 1 month label Jan 4, 2024

blathers-crl bot mentioned this issue Jan 4, 2024

release-23.2.0-rc: concurrency: quieten KVNemesis failure because of validation #117329

Merged

This was referenced Jan 17, 2024

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #117855

Closed

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #118014

Closed

cockroach-teamcity mentioned this issue Jan 25, 2024

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #118301

Closed

github-actions bot added the no-test-failure-activity label Mar 4, 2024

github-actions bot removed the no-test-failure-activity label Mar 11, 2024

This was referenced Mar 31, 2024

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #121426

Closed

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #121748

Closed

arulajmani mentioned this issue Apr 10, 2024

concurrency: add lock table verification for lock promotion state #122056

Merged

github-actions bot added the no-test-failure-activity label Apr 15, 2024

github-actions bot added the X-stale label Apr 22, 2024

github-actions bot closed this as completed Apr 22, 2024

github-project-automation bot added this to KV Aug 28, 2024

github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

cockroach-teamcity commented Dec 5, 2023 •

edited by cockroach-jira-scripts

Loading

erikgrinaker commented Dec 5, 2023

cockroach-teamcity commented Dec 5, 2023

arulajmani commented Dec 5, 2023

arulajmani commented Dec 5, 2023

cockroach-teamcity commented Dec 7, 2023

cockroach-teamcity commented Dec 11, 2023

cockroach-teamcity commented Dec 12, 2023

cockroach-teamcity commented Dec 13, 2023

github-actions bot commented Mar 4, 2024

cockroach-teamcity commented Mar 8, 2024

github-actions bot commented Apr 15, 2024

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

Comments

cockroach-teamcity commented Dec 5, 2023 • edited by cockroach-jira-scripts Loading

erikgrinaker commented Dec 5, 2023

cockroach-teamcity commented Dec 5, 2023

arulajmani commented Dec 5, 2023

arulajmani commented Dec 5, 2023

cockroach-teamcity commented Dec 7, 2023

cockroach-teamcity commented Dec 11, 2023

cockroach-teamcity commented Dec 12, 2023

cockroach-teamcity commented Dec 13, 2023

github-actions bot commented Mar 4, 2024

cockroach-teamcity commented Mar 8, 2024

github-actions bot commented Apr 15, 2024

cockroach-teamcity commented Dec 5, 2023 •

edited by cockroach-jira-scripts

Loading