Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17 (1) Wraps... #126072

cockroach-sentry · 2024-06-23T07:49:12Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5525094099/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17
(1)
Wraps: (2)
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:385
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:272
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:278
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*projectSetProcessor).nextInputRow
  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/project_set.go:155
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*projectSetProcessor).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/project_set.go:291
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:239
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:157
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*tupleProjOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/tuple_proj_op.go:59
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:427
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:272
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:278
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*noopProcessor).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/noop.go:88
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashAggregator).accumulateRows
  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:424
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*hashAggregator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:702
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:239
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:254
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:328
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1895
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1627
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1630
Wraps: (4)
Wraps: (5) tags: [n1,client=192.168.50.34:37710,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=1,distsql.appname=×,distsql.txn=×,streamID=×,received-error]
Wraps: (6) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.appname=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (7)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (8)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:779
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:551
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1206
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1306
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:849
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Ru...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/distsql_running.go

Lines 1629 to 1631 in b95f222

cockroach/pkg/sql/distsql_running.go

Lines 1626 to 1628 in b95f222

    
           defer cleanup() 
        
           dsp.PlanAndRun( 
        
           	ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1894 to 1896 in b95f222

    
           	recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows) 
        
           	dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn) 
        
           }

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in b95f222

    
           	noWait := planCtx.getPortalPauseInfo() != nil 
        
           	flow.Run(ctx, noWait) 
        
           }

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 327 to 329 in b95f222

    
           	log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine") 
        
           	f.batchFlowCoordinator.Run(ctx) 
        
           }

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 285 to 287 in b95f222

    
           for status == execinfra.NeedMoreRows { 
        
           	err := f.next() 
        
           	if err != nil {

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 253 to 255 in b95f222

    
           func (f *BatchFlowCoordinator) next() error { 
        
           	return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in b95f222

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 249 to 251 in b95f222

    
           func (f *BatchFlowCoordinator) nextAdapter() { 
        
           	f.batch = f.input.Root.Next() 
        
           }

cockroach/pkg/sql/colexec/columnarizer.go

Lines 238 to 240 in b95f222

    
           for batchDone := false; !batchDone; { 
        
           	row, meta := c.input.Next() 
        
           	if meta != nil {

cockroach/pkg/sql/rowexec/aggregator.go

Lines 701 to 703 in b95f222

    
           case aggAccumulating: 
        
           	ag.runningState, row, meta = ag.accumulateRows() 
        
           case aggEmittingRows:

cockroach/pkg/sql/rowexec/aggregator.go

Lines 423 to 425 in b95f222

    
           for { 
        
           	row, meta := ag.input.Next() 
        
           	if meta != nil {

cockroach/pkg/sql/rowexec/noop.go

Lines 87 to 89 in b95f222

    
           for n.State == execinfra.StateRunning { 
        
           	row, meta := n.input.Next()

cockroach/pkg/sql/colexec/materializer.go

Lines 277 to 279 in b95f222

    
           for m.State == execinfra.StateRunning { 
        
           	if err := colexecerror.CatchVectorizedRuntimeError(m.nextAdapter); err != nil { 
        
           		m.MoveToDraining(err)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in b95f222

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colexec/materializer.go

Lines 271 to 273 in b95f222

    
           func (m *Materializer) nextAdapter() { 
        
           	m.outputRow = m.next() 
        
           }

cockroach/pkg/sql/colexec/materializer.go

Lines 246 to 248 in b95f222

    
           // Get a fresh batch. 
        
           m.batch = m.input.Next() 
        
           if m.batch.Length() == 0 {

cockroach/pkg/sql/colexecop/operator.go

Lines 426 to 428 in b95f222

    
           func (n *noopOperator) Next() coldata.Batch { 
        
           	return n.Input.Next() 
        
           }

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 123 to 125 in b95f222

    
           func (d *simpleProjectOp) Next() coldata.Batch { 
        
           	batch := d.Input.Next() 
        
           	if batch.Length() == 0 {

cockroach/pkg/sql/colexec/tuple_proj_op.go

Lines 58 to 60 in b95f222

    
           func (t *tupleProjOp) Next() coldata.Batch { 
        
           	batch := t.Input.Next() 
        
           	n := batch.Length()

cockroach/pkg/sql/colexec/colexecutils/operator.go

Lines 156 to 158 in b95f222

    
           func (e *vectorTypeEnforcer) Next() coldata.Batch { 
        
           	b := e.Input.Next() 
        
           	if b.Length() == 0 {

cockroach/pkg/sql/colexec/columnarizer.go

Lines 238 to 240 in b95f222

    
           for batchDone := false; !batchDone; { 
        
           	row, meta := c.input.Next() 
        
           	if meta != nil {

cockroach/pkg/sql/rowexec/project_set.go

Lines 290 to 292 in b95f222

    
           // Read the row from the source. 
        
           row, meta, err := ps.nextInputRow() 
        
           if meta != nil {

cockroach/pkg/sql/rowexec/project_set.go

Lines 154 to 156 in b95f222

    
           ) { 
        
           	row, meta := ps.input.Next() 
        
           	if row == nil {

cockroach/pkg/sql/colexec/materializer.go

Lines 277 to 279 in b95f222

    
           for m.State == execinfra.StateRunning { 
        
           	if err := colexecerror.CatchVectorizedRuntimeError(m.nextAdapter); err != nil { 
        
           		m.MoveToDraining(err)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in b95f222

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colexec/materializer.go

Lines 271 to 273 in b95f222

    
           func (m *Materializer) nextAdapter() { 
        
           	m.outputRow = m.next() 
        
           }

cockroach/pkg/sql/colexec/materializer.go

Lines 246 to 248 in b95f222

    
           // Get a fresh batch. 
        
           m.batch = m.input.Next() 
        
           if m.batch.Length() == 0 {

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 384 to 386 in b95f222

    
           if receivedErr != nil { 
        
           	colexecerror.ExpectedError(receivedErr) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in b95f222

    
           func ExpectedError(err error) { 
        
           	panic(newNotInternalError(err)) 
        
           }

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in b95f222

    
           } 
        
           err := logcrash.PanicAsError(0, panicObj) 
        
           log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

cockroach/pkg/kv/db.go

Lines 1011 to 1013 in b95f222

    
           br, pErr := sender.Send(ctx, ba) 
        
           if pErr != nil {

cockroach/pkg/kv/db.go

Lines 223 to 225 in b95f222

    
           br, pErr := s.wrapped.Send(ctx, ba) 
        
           if _, ok := pErr.GetDetail().(*kvpb.OpRequiresTxnError); !ok {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 932 to 934 in b95f222

    
           } else { 
        
           	rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */) 
        
           }

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1311 to 1313 in b95f222

    
           if !ri.NeedAnother(rs) { 
        
           	resp := ds.sendPartialBatch( 
        
           		ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), nil, /* positions */

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1740 to 1742 in b95f222

    
           prevTok = routingTok 
        
           reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2237 to 2239 in b95f222

    
           } 
        
           br, err = transport.SendNext(ctx, ba) 
        
           ds.maybeIncrementErrCounters(br, err)

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in b95f222

    
           	} 
        
           	return gt.sendBatch(ctx, r.NodeID, iface, ba) 
        
           }

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 210 to 212 in b95f222

    
           log.VEvent(ctx, 2, "sending batch request") 
        
           reply, err := iface.Batch(ctx, ba) 
        
           log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in b95f222

    
           if localRequest { 
        
           	return invoker(ctx, method, req, resp, cc, opts...) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in b95f222

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in b95f222

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1305 to 1307 in b95f222


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1205 to 1207 in b95f222

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L550-L552
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L778-L780

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in b95f222

    
           	log.VEventf(ctx, 2, "recovery error: %v", res.Err) 
        
           	return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery") 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in b95f222

    
           if localRequest { 
        
           	return invoker(ctx, method, req, resp, cc, opts...) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in b95f222

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in b95f222

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1305 to 1307 in b95f222


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1205 to 1207 in b95f222

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go

Lines 132 to 134 in b95f222

    
           if was, is := args.Txn.WriteTimestamp, reply.RecoveredTxn.WriteTimestamp; was != is { 
        
           	return result.Result{}, errors.AssertionFailedf( 
        
           		"programming error: timestamp change by implicitly committed transaction: %v->%v", was, is,

pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1630
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1627
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1895
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 328
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 286
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 254
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 250
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 239
pkg/sql/rowexec/aggregator.go in pkg/sql/rowexec.(*hashAggregator).Next at line 702
pkg/sql/rowexec/aggregator.go in pkg/sql/rowexec.(*hashAggregator).accumulateRows at line 424
pkg/sql/rowexec/noop.go in pkg/sql/rowexec.(*noopProcessor).Next at line 88
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 278
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 272
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 247
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 427
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 124
pkg/sql/colexec/tuple_proj_op.go in pkg/sql/colexec.(*tupleProjOp).Next at line 59
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 157
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 239
pkg/sql/rowexec/project_set.go in pkg/sql/rowexec.(*projectSetProcessor).Next at line 291
pkg/sql/rowexec/project_set.go in pkg/sql/rowexec.(*projectSetProcessor).nextInputRow at line 155
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 278
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 272
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 247
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 385
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1012
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 933
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1312
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1741
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2238
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1306
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1206
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 551
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 779
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1306
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1206
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 176
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 123
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 844
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 756
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 442
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 668
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 708
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 294
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 526
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 133

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17 (1) Wraps... #126072

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17 (1) Wraps... #126072

cockroach-sentry commented Jun 23, 2024 •

edited by cockroach-jira-scripts

Loading

blathers-crl bot commented Jun 23, 2024

yuzefovich commented Jun 24, 2024

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17 (1) Wraps... #126072

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1719128878.314537702,2->1719128869.089208853,17 (1) Wraps... #126072

Comments

cockroach-sentry commented Jun 23, 2024 • edited by cockroach-jira-scripts Loading

Tags

blathers-crl bot commented Jun 23, 2024

yuzefovich commented Jun 24, 2024

cockroach-sentry commented Jun 23, 2024 •

edited by cockroach-jira-scripts

Loading