kvserver: v22.1.5: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request? #86894

cockroach-teamcity · 2022-08-25T18:04:59Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3531799696/?referrer=webhooks_plugin

Panic message:

concurrency_manager.go:440: log.Fatal: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request?
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).HandleWriterIntentError
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:440
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleWriteIntentError
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:723
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:505
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithoutRangeID
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:170
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:100
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:197
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:191
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
| github.com/cockroachdb/cockroach/pkg/server/node.go:1006
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| github.com/cockroachdb/cockroach/pkg/server/node.go:989
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| github.com/cockroachdb/cockroach/pkg/server/node.go:1058
| github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:609
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:209
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:191
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2062
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1608
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1378
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:831
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:242
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:177
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:290
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:532
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| github.com/cockroachdb/cockroach/pkg/kv/db.go:985
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:1124
| github.com/cockroachdb/cockroach/pkg/sql/row.makeKVBatchFetcherDefaultSendFunc.func1
| github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:181
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:347
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:531
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:209
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher/cfetcher.go:699
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_scan.go:107
Wraps: (2) secondary error attachment
| ×
| (1) ×
| Error types: (1) *roachpb.WriteIntentError
Wraps: (3) log.Fatal: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request?
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request?
*secondary.withSecondaryError
concurrency_manager.go:440: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go

Lines 439 to 441 in a30a663

    
           if g.ltg == nil { 
        
           	log.Fatalf(ctx, "cannot handle WriteIntentError %v for request without "+ 
        
           		"lockTableGuard; were lock spans declared for this request?", t)

in pkg/kv/kvserver/concurrency.(*managerImpl).HandleWriterIntentError
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L722-L724 in pkg/kv/kvserver.(*Replica).handleWriteIntentError
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L504-L506 in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L169-L171 in pkg/kv/kvserver.(*Replica).sendWithoutRangeID
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L99-L101 in pkg/kv/kvserver.(*Replica).Send
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L196-L198 in pkg/kv/kvserver.(*Store).Send
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L190-L192 in pkg/kv/kvserver.(*Stores).Send

cockroach/pkg/server/node.go

Lines 1005 to 1007 in a30a663

    
           var pErr *roachpb.Error 
        
           br, pErr = n.stores.Send(ctx, *args) 
        
           if pErr != nil {

in pkg/server.(*Node).batchInternal.func1

cockroach/pkg/util/stop/stopper.go

Lines 343 to 345 in a30a663

    
           	return f(ctx) 
        
           }

in pkg/util/stop.(*Stopper).RunTaskWithErr

cockroach/pkg/server/node.go

Lines 988 to 990 in a30a663

    
           var br *roachpb.BatchResponse 
        
           if err := n.stopper.RunTaskWithErr(ctx, "node.Node: batch", func(ctx context.Context) error { 
        
           	var reqSp spanForRequest

in pkg/server.(*Node).batchInternal

cockroach/pkg/server/node.go

Lines 1057 to 1059 in a30a663


	br, err := n.batchInternal(ctx, tenantID, args)

in pkg/server.(*Node).Batch
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/rpc/pkg/rpc/context.go#L608-L610 in pkg/rpc.internalClientAdapter.Batch

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 208 to 210 in a30a663

    
           } 
        
           reply, err := iface.Batch(ctx, &ba) 
        
           // If we queried a remote node, perform extra validation and

in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 190 to 192 in a30a663

    
           	ba.Replica = r 
        
           	return gt.sendBatch(ctx, r.NodeID, iface, ba) 
        
           }

in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2061 to 2063 in a30a663

    
           } 
        
           br, err = transport.SendNext(ctx, ba) 
        
           ds.maybeIncrementErrCounters(br, err)

in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1607 to 1609 in a30a663

    
           prevTok = routingTok 
        
           reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1377 to 1379 in a30a663

    
           } else { 
        
           	resp := ds.sendPartialBatch( 
        
           		ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), true, /* needsTruncate */

in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 830 to 832 in a30a663

    
           } else { 
        
           	rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */) 
        
           }

in pkg/kv/kvclient/kvcoord.(*DistSender).Send

cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go

Lines 81 to 83 in a30a663

    
           	defer gs.mu.Lock() 
        
           	return gs.wrapped.Send(ctx, ba) 
        
           }

in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go

Lines 241 to 243 in a30a663

    
           } 
        
           br, pErr := sr.wrapped.SendLocked(ctx, ba)

in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go

Lines 176 to 178 in a30a663

    
           // Send through wrapped lockedSender. Unlocks while sending then re-locks. 
        
           br, pErr := sr.sendLockedWithRefreshAttempts(ctx, ba, sr.maxRefreshAttempts()) 
        
           if pErr != nil {

in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go

Lines 289 to 291 in a30a663

    
           // Send through wrapped lockedSender. Unlocks while sending then re-locks. 
        
           br, pErr := tp.wrapped.SendLocked(ctx, ba)

in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go

Lines 104 to 106 in a30a663

    
           	return s.wrapped.SendLocked(ctx, ba) 
        
           }

in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 531 to 533 in a30a663

    
           // Send the command through the txnInterceptor stack. 
        
           br, pErr := tc.interceptorStack[0].SendLocked(ctx, ba)

in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send

cockroach/pkg/kv/db.go

Lines 984 to 986 in a30a663

    
           br, pErr := sender.Send(ctx, ba) 
        
           if pErr != nil {

in pkg/kv.(*DB).sendUsingSender

cockroach/pkg/kv/txn.go

Lines 1123 to 1125 in a30a663

    
           txn.mu.Unlock() 
        
           br, pErr := txn.db.sendUsingSender(ctx, ba, sender) 
        
           if pErr == nil {

in pkg/kv.(*Txn).Send

cockroach/pkg/sql/row/kv_batch_fetcher.go

Lines 180 to 182 in a30a663

    
           ) (*roachpb.BatchResponse, error) { 
        
           	res, err := txn.Send(ctx, ba) 
        
           	if err != nil {

in pkg/sql/row.makeKVBatchFetcherDefaultSendFunc.func1

cockroach/pkg/sql/row/kv_batch_fetcher.go

Lines 346 to 348 in a30a663

    
           br, err := f.sendFn(ctx, ba) 
        
           if err != nil {

in pkg/sql/row.(*txnKVFetcher).fetch

cockroach/pkg/sql/row/kv_batch_fetcher.go

Lines 530 to 532 in a30a663

    
           // We have more work to do. Ask the KV layer to continue where it left off. 
        
           if err := f.fetch(ctx); err != nil { 
        
           	return false, nil, nil, err

in pkg/sql/row.(*txnKVFetcher).nextBatch

cockroach/pkg/sql/row/kv_fetcher.go

Lines 208 to 210 in a30a663

    
           ok, f.kvs, f.batchResponse, err = f.nextBatch(ctx) 
        
           if err != nil || !ok {

in pkg/sql/row.(*KVFetcher).NextKV

cockroach/pkg/sql/colfetcher/cfetcher.go

Lines 698 to 700 in a30a663

    
           case stateInitFetch: 
        
           	moreKVs, kv, finalReferenceToBatch, err := cf.fetcher.NextKV(ctx, cf.mvccDecodeStrategy) 
        
           	if err != nil {

in pkg/sql/colfetcher.(*cFetcher).NextBatch

cockroach/pkg/sql/colfetcher/colbatch_scan.go

Lines 106 to 108 in a30a663

    
           func (s *ColBatchScan) Next() coldata.Batch { 
        
           	bat, err := s.cf.NextBatch(s.Ctx) 
        
           	if err != nil {

in pkg/sql/colfetcher.(*ColBatchScan).Next

pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).HandleWriterIntentError at line 440
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleWriteIntentError at line 723
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 505
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).sendWithoutRangeID at line 170
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).Send at line 100
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).Send at line 197
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).Send at line 191
pkg/server/node.go in pkg/server.(*Node).batchInternal.func1 at line 1006
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 989
pkg/server/node.go in pkg/server.(*Node).Batch at line 1058
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 609
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 209
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 191
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2062
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1608
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1378
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 831
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked at line 82
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts at line 242
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 177
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 290
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 105
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 532
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 985
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1124
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.makeKVBatchFetcherDefaultSendFunc.func1 at line 181
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 347
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 531
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 209
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 699
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 107

Tag	Value
Cockroach Release	`v22.1.5`
Cockroach SHA:	`a30a663`
Platform	windows amd64
Distribution	CCL
Environment	v22.1.5
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-18991

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-08-30T17:05:37Z

seems like a dup of #79630, but 22.1.5 should contain the fix for that issue, cc @nvanbenschoten

nvanbenschoten · 2023-01-03T21:56:09Z

This is on v22.1 and we haven't seen the issue since (according to Sentry). Closing.

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Aug 25, 2022

blathers-crl bot added the T-kv KV Team label Aug 30, 2022

exalate-issue-sync bot assigned AlexTalks Sep 19, 2022

nvanbenschoten closed this as completed Jan 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvserver: v22.1.5: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request? #86894

kvserver: v22.1.5: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request? #86894

cockroach-teamcity commented Aug 25, 2022 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Aug 30, 2022

nvanbenschoten commented Jan 3, 2023

kvserver: v22.1.5: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request? #86894

kvserver: v22.1.5: cannot handle WriteIntentError × for request without lockTableGuard; were lock spans declared for this request? #86894

Comments

cockroach-teamcity commented Aug 25, 2022 • edited by cockroach-jira-scripts Loading

yuzefovich commented Aug 30, 2022

nvanbenschoten commented Jan 3, 2023

cockroach-teamcity commented Aug 25, 2022 •

edited by cockroach-jira-scripts

Loading