Sentry: engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/× (1) attached stack trace -- stack trace: | github.com/cockroac... #120459

cockroach-sentry · 2024-03-14T02:40:25Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5064538438/?referrer=webhooks_plugin

Panic Message:

engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/×
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:383
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:427
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:254
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:328
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1895
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1627
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1630
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2153
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1707
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1036
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3083
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2183
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2188
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2105
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:902
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2)
Wraps: (3) tags: [n2,client=15.204.46.111:56338,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=2,distsql.appname=×,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.appname=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (5)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (6)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:779
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:551
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1205
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1305
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:849
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:249
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorTo...

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 8d065df

    
           reservedOwned = false // We're about to pass ownership away. 
        
           retErr = sqlServer.ServeConn( 
        
           	ctx,

cockroach/pkg/sql/conn_executor.go

Lines 901 to 903 in 8d065df

    
           	}(ctx, h) 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

cockroach/pkg/sql/conn_executor.go

Lines 2104 to 2106 in 8d065df

    
           var err error 
        
           if err = ex.execCmd(); err != nil { 
        
           	// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2187 to 2189 in 8d065df

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2182 to 2184 in 8d065df

    
           		tcmd.LastInBatch || !implicitTxnForBatch) 
        
           ev, payload, err = ex.execStmt( 
        
           	ctx, tcmd.Statement, nil /* portal */, nil /* pinfo */, stmtRes, canAutoCommit,

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 8d065df

    
           } 
        
           err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3082 to 3084 in 8d065df

    
           } else { 
        
           	err = op(ctx) 
        
           }

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 8d065df

    
           err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit) 
        
           	return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1035 to 1037 in 8d065df

    
           if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1706 to 1708 in 8d065df

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err = ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 2152 to 2154 in 8d065df

    
           	} 
        
           	err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory) 
        
           }

cockroach/pkg/sql/distsql_running.go

Lines 1629 to 1631 in 8d065df

cockroach/pkg/sql/distsql_running.go

Lines 1626 to 1628 in 8d065df

    
           defer cleanup() 
        
           dsp.PlanAndRun( 
        
           	ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1894 to 1896 in 8d065df

    
           	recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows) 
        
           	dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn) 
        
           }

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in 8d065df

    
           	noWait := planCtx.getPortalPauseInfo() != nil 
        
           	flow.Run(ctx, noWait) 
        
           }

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 327 to 329 in 8d065df

    
           	log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine") 
        
           	f.batchFlowCoordinator.Run(ctx) 
        
           }

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 285 to 287 in 8d065df

    
           for status == execinfra.NeedMoreRows { 
        
           	err := f.next() 
        
           	if err != nil {

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 253 to 255 in 8d065df

    
           func (f *BatchFlowCoordinator) next() error { 
        
           	return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 8d065df

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 249 to 251 in 8d065df

    
           func (f *BatchFlowCoordinator) nextAdapter() { 
        
           	f.batch = f.input.Root.Next() 
        
           }

cockroach/pkg/sql/colflow/stats.go

Lines 125 to 127 in 8d065df

    
           // execution time (e.g. in the statement bundle). 
        
           err := colexecerror.CatchVectorizedRuntimeError(bic.next) 
        
           bic.stopwatch.Stop()

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 8d065df

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colflow/stats.go

Lines 117 to 119 in 8d065df

    
           func (bic *batchInfoCollector) next() { 
        
           	bic.batch = bic.Input.Next() 
        
           }

cockroach/pkg/sql/colexecop/operator.go

Lines 426 to 428 in 8d065df

    
           func (n *noopOperator) Next() coldata.Batch { 
        
           	return n.Input.Next() 
        
           }

cockroach/pkg/sql/colflow/stats.go

Lines 125 to 127 in 8d065df

    
           // execution time (e.g. in the statement bundle). 
        
           err := colexecerror.CatchVectorizedRuntimeError(bic.next) 
        
           bic.stopwatch.Stop()

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 8d065df

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colflow/stats.go

Lines 117 to 119 in 8d065df

    
           func (bic *batchInfoCollector) next() { 
        
           	bic.batch = bic.Input.Next() 
        
           }

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 382 to 384 in 8d065df

    
           if receivedErr != nil { 
        
           	colexecerror.ExpectedError(receivedErr) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in 8d065df

    
           func ExpectedError(err error) { 
        
           	panic(newNotInternalError(err)) 
        
           }

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in 8d065df

    
           } 
        
           err := logcrash.PanicAsError(0, panicObj) 
        
           log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

cockroach/pkg/kv/db.go

Lines 1011 to 1013 in 8d065df

    
           br, pErr := sender.Send(ctx, ba) 
        
           if pErr != nil {

cockroach/pkg/kv/db.go

Lines 223 to 225 in 8d065df

    
           br, pErr := s.wrapped.Send(ctx, ba) 
        
           if _, ok := pErr.GetDetail().(*kvpb.OpRequiresTxnError); !ok {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 932 to 934 in 8d065df

    
           } else { 
        
           	rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */) 
        
           }

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1311 to 1313 in 8d065df

    
           if !ri.NeedAnother(rs) { 
        
           	resp := ds.sendPartialBatch( 
        
           		ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), nil, /* positions */

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1740 to 1742 in 8d065df

    
           prevTok = routingTok 
        
           reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2237 to 2239 in 8d065df

    
           } 
        
           br, err = transport.SendNext(ctx, ba) 
        
           ds.maybeIncrementErrCounters(br, err)

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in 8d065df

    
           	} 
        
           	return gt.sendBatch(ctx, r.NodeID, iface, ba) 
        
           }

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 209 to 211 in 8d065df

    
           } 
        
           reply, err := iface.Batch(ctx, ba) 
        
           // If we queried a remote node, perform extra validation.

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in 8d065df

    
           if localRequest { 
        
           	return invoker(ctx, method, req, resp, cc, opts...) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 8d065df

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 8d065df

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 8d065df


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 8d065df

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L550-L552
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L778-L780

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in 8d065df

    
           	log.VEventf(ctx, 2, "recovery error: %v", res.Err) 
        
           	return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery") 
        
           }

GOROOT/src/runtime/asm_amd64.s#L1593-L1595
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1703-L1705
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1335-L1337
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10088-L10090
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1153-L1155
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 8d065df

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L250-L252
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/auth.go#L104-L106
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L283-L285
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 8d065df

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10086-L10088

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 8d065df


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 8d065df

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

cockroach/pkg/kv/kvserver/batcheval/cmd_query_intent.go

Lines 78 to 80 in 8d065df

    
           // With prefix set to true there should be at most one result. 
        
           intent, err := storage.GetIntent(reader, args.Key) 
        
           if err != nil {

cockroach/pkg/storage/engine.go

Lines 1310 to 1312 in 8d065df

    
           	} 
        
           	return nil, errors.AssertionFailedf("unexpected additional key found %v while looking for %v", engineKey, key) 
        
           }

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 902
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2105
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2188
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2183
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3083
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1036
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1707
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2153
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1630
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1627
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1895
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 328
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 286
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 254
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 250
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 126
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 118
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 427
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 126
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 118
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 383
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1012
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 933
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1312
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1741
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2238
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 210
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 551
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 779
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1704
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1336
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler at line 10089
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1 at line 1154
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler.func1 at line 10087
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 181
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 124
pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes at line 457
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 294
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 528
pkg/kv/kvserver/batcheval/cmd_query_intent.go in pkg/kv/kvserver/batcheval.QueryIntent at line 79
pkg/storage/engine.go in pkg/storage.GetIntent at line 1311

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry: engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/× (1) attached stack trace -- stack trace: | github.com/cockroac... #120459

Sentry: engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/× (1) attached stack trace -- stack trace: | github.com/cockroac... #120459

cockroach-sentry commented Mar 14, 2024 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Mar 17, 2024

Sentry: engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/× (1) attached stack trace -- stack trace: | github.com/cockroac... #120459

Sentry: engine.go:1311: failed indeterminate commit recovery: unexpected additional key found × while looking for /Table/287/1/×/× (1) attached stack trace -- stack trace: | github.com/cockroac... #120459

Comments

cockroach-sentry commented Mar 14, 2024 • edited by cockroach-jira-scripts Loading

Tags

yuzefovich commented Mar 17, 2024

cockroach-sentry commented Mar 14, 2024 •

edited by cockroach-jira-scripts

Loading