Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0 (1) attach... #120183

cockroach-sentry · 2024-03-10T13:45:24Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5051764110/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:383
  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:427
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:254
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:328
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1895
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1627
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1630
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2153
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1707
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1036
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3083
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:247
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2288
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2290
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2105
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:902
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2)
Wraps: (3) tags: [n2,client=15.204.46.111:50594,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=2,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (5)
Wraps: (6)
Wraps: (7)
Wraps: (8)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (9)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:779
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:551
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1205
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1305
  | github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10087
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:249
  | google.golang.org/grpc.chainUnaryInterceptors.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154
  | github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10089
  | google.golang.org/grpc.(*Server).processUnaryRPC
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336
  | google.golang.org/grpc.(*Server).handleStream
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704
  | google.golang.org/grpc.(*Server).serveStreams.func1.2
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
  | runtime.goex...

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 8d065df

    
           reservedOwned = false // We're about to pass ownership away. 
        
           retErr = sqlServer.ServeConn( 
        
           	ctx,

cockroach/pkg/sql/conn_executor.go

Lines 901 to 903 in 8d065df

    
           	}(ctx, h) 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

cockroach/pkg/sql/conn_executor.go

Lines 2104 to 2106 in 8d065df

    
           var err error 
        
           if err = ex.execCmd(); err != nil { 
        
           	// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2289 to 2291 in 8d065df

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2287 to 2289 in 8d065df

    
           canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync 
        
           ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit) 
        
           return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 246 to 248 in 8d065df

    
           } 
        
           ev, payload, retErr = ex.execStmt(ctx, portal.Stmt.Statement, &portal, pinfo, stmtRes, canAutoCommit) 
        
           // For a non-pausable portal, it is considered exhausted regardless of the

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 8d065df

    
           } 
        
           err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3082 to 3084 in 8d065df

    
           } else { 
        
           	err = op(ctx) 
        
           }

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 8d065df

    
           err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit) 
        
           	return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1035 to 1037 in 8d065df

    
           if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1706 to 1708 in 8d065df

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err = ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 2152 to 2154 in 8d065df

    
           	} 
        
           	err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory) 
        
           }

cockroach/pkg/sql/distsql_running.go

Lines 1629 to 1631 in 8d065df

cockroach/pkg/sql/distsql_running.go

Lines 1626 to 1628 in 8d065df

    
           defer cleanup() 
        
           dsp.PlanAndRun( 
        
           	ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1894 to 1896 in 8d065df

    
           	recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows) 
        
           	dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn) 
        
           }

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in 8d065df

    
           	noWait := planCtx.getPortalPauseInfo() != nil 
        
           	flow.Run(ctx, noWait) 
        
           }

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 327 to 329 in 8d065df

    
           	log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine") 
        
           	f.batchFlowCoordinator.Run(ctx) 
        
           }

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 285 to 287 in 8d065df

    
           for status == execinfra.NeedMoreRows { 
        
           	err := f.next() 
        
           	if err != nil {

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 253 to 255 in 8d065df

    
           func (f *BatchFlowCoordinator) next() error { 
        
           	return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 8d065df

    
           }() 
        
           operation() 
        
           return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 249 to 251 in 8d065df

    
           func (f *BatchFlowCoordinator) nextAdapter() { 
        
           	f.batch = f.input.Root.Next() 
        
           }

cockroach/pkg/sql/colexecop/operator.go

Lines 426 to 428 in 8d065df

    
           func (n *noopOperator) Next() coldata.Batch { 
        
           	return n.Input.Next() 
        
           }

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 382 to 384 in 8d065df

    
           if receivedErr != nil { 
        
           	colexecerror.ExpectedError(receivedErr) 
        
           }

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in 8d065df

    
           func ExpectedError(err error) { 
        
           	panic(newNotInternalError(err)) 
        
           }

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in 8d065df

    
           } 
        
           err := logcrash.PanicAsError(0, panicObj) 
        
           log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

GOROOT/src/runtime/asm_amd64.s#L1593-L1595
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1703-L1705
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1335-L1337
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10088-L10090
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1153-L1155
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 8d065df

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L250-L252
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/auth.go#L104-L106
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L283-L285
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 8d065df

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go#L10086-L10088

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 8d065df


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 8d065df

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L550-L552
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L778-L780

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in 8d065df

    
           	log.VEventf(ctx, 2, "recovery error: %v", res.Err) 
        
           	return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery") 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in 8d065df

    
           if localRequest { 
        
           	return invoker(ctx, method, req, resp, cc, opts...) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 8d065df

    
           	return f(ctx) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 8d065df

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 8d065df


	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 8d065df

    
           var pErr *kvpb.Error 
        
           br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args) 
        
           if pErr != nil {

cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go

Lines 132 to 134 in 8d065df

    
           if was, is := args.Txn.WriteTimestamp, reply.RecoveredTxn.WriteTimestamp; was != is { 
        
           	return result.Result{}, errors.AssertionFailedf( 
        
           		"programming error: timestamp change by implicitly committed transaction: %v->%v", was, is,

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 902
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2105
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2290
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 2288
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 247
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3083
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1036
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1707
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2153
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1630
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1627
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1895
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 328
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 286
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 254
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 250
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 427
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 383
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1704
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1336
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler at line 10089
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1 at line 1154
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_Batch_Handler.func1 at line 10087
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 551
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 779
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 176
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 123
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 844
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 756
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 442
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 668
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 708
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 294
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 526
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 133

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0 (1) attach... #120183

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0 (1) attach... #120183

cockroach-sentry commented Mar 10, 2024 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Mar 17, 2024

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0 (1) attach... #120183

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1710078299.270715885,2->1710078282.130892964,0 (1) attach... #120183

Comments

cockroach-sentry commented Mar 10, 2024 • edited by cockroach-jira-scripts Loading

Tags

yuzefovich commented Mar 17, 2024

cockroach-sentry commented Mar 10, 2024 •

edited by cockroach-jira-scripts

Loading