Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1706233945.398780464,2->1706233938.623665726,0 (1) attach... #118339

Closed
cockroach-sentry opened this issue Jan 26, 2024 · 1 comment
Closed

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1706233945.398780464,2->1706233938.623665726,0 (1) attach... #118339

cockroach-sentry opened this issue Jan 26, 2024 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Jan 26, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4916787127/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1706233945.398780464,2->1706233938.623665726,0
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:383
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*OrderedSynchronizer).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/ordered_synchronizer.eg.go:119
  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:427
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*limitOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/limit.go:47
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:254
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:286
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:324
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1913
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1645
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1648
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2155
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1709
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1036
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3086
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:247
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2287
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2289
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2104
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:901
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2)
Wraps: (3) tags: [n2,client=185.213.174.235:52430,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=2,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.txn=×,streamID=×,sent-error=×]
Wraps: (5)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (6)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:779
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:551
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1205
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1305
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:841
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:276
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:243
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:851
  | ...
Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 28b2e76

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(
ctx,

cockroach/pkg/sql/conn_executor.go

Lines 900 to 902 in 28b2e76

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}

cockroach/pkg/sql/conn_executor.go

Lines 2103 to 2105 in 28b2e76

var err error
if err = ex.execCmd(); err != nil {
// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2288 to 2290 in 28b2e76

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2286 to 2288 in 28b2e76

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 246 to 248 in 28b2e76

}
ev, payload, retErr = ex.execStmt(ctx, portal.Stmt.Statement, &portal, pinfo, stmtRes, canAutoCommit)
// For a non-pausable portal, it is considered exhausted regardless of the

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 28b2e76

}
err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3085 to 3087 in 28b2e76

} else {
err = op(ctx)
}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 28b2e76

err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1035 to 1037 in 28b2e76

if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1708 to 1710 in 28b2e76

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 2154 to 2156 in 28b2e76

}
err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
}

cockroach/pkg/sql/distsql_running.go

Lines 1647 to 1649 in 28b2e76

)
}()

cockroach/pkg/sql/distsql_running.go

Lines 1644 to 1646 in 28b2e76

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1912 to 1914 in 28b2e76

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in 28b2e76

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 323 to 325 in 28b2e76

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 285 to 287 in 28b2e76

for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 253 to 255 in 28b2e76

func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 28b2e76

}()
operation()
return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 249 to 251 in 28b2e76

func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}

cockroach/pkg/sql/colexec/limit.go

Lines 46 to 48 in 28b2e76

}
bat := c.Input.Next()
length := bat.Length()

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 123 to 125 in 28b2e76

func (d *simpleProjectOp) Next() coldata.Batch {
batch := d.Input.Next()
if batch.Length() == 0 {

cockroach/pkg/sql/colexecop/operator.go

Lines 426 to 428 in 28b2e76

func (n *noopOperator) Next() coldata.Batch {
return n.Input.Next()
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/bazel-out/k8-opt/bin/pkg/sql/colexec/ordered_synchronizer.eg.go#L118-L120

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 382 to 384 in 28b2e76

if receivedErr != nil {
colexecerror.ExpectedError(receivedErr)
}

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in 28b2e76

func ExpectedError(err error) {
panic(newNotInternalError(err))
}

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in 28b2e76

}
err := logcrash.PanicAsError(0, panicObj)
log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

cockroach/pkg/kv/db.go

Lines 1005 to 1007 in 28b2e76

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/db.go

Lines 223 to 225 in 28b2e76

br, pErr := s.wrapped.Send(ctx, ba)
if _, ok := pErr.GetDetail().(*kvpb.OpRequiresTxnError); !ok {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 899 to 901 in 28b2e76

} else {
rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */)
}

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1278 to 1280 in 28b2e76

if !ri.NeedAnother(rs) {
resp := ds.sendPartialBatch(
ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), nil, /* positions */

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1707 to 1709 in 28b2e76

prevTok = routingTok
reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2181 to 2183 in 28b2e76

}
br, err = transport.SendNext(ctx, ba)
ds.maybeIncrementErrCounters(br, err)

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in 28b2e76

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 209 to 211 in 28b2e76

}
reply, err := iface.Batch(ctx, ba)
// If we queried a remote node, perform extra validation.

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L1043-L1045
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L920-L922
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L1035-L1037

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 226 to 228 in 28b2e76

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L850-L852
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L240-L242

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 28b2e76

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L242-L244
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L275-L277
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 95 to 97 in 28b2e76

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L840-L842

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 28b2e76

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 28b2e76

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L550-L552
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L778-L780

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in 28b2e76

log.VEventf(ctx, 2, "recovery error: %v", res.Err)
return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery")
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L1043-L1045
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L920-L922
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L1035-L1037

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 226 to 228 in 28b2e76

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L850-L852
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L240-L242

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 28b2e76

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L242-L244
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L275-L277
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L951-L953

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 95 to 97 in 28b2e76

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/rpc/pkg/rpc/context.go#L840-L842

cockroach/pkg/server/node.go

Lines 1304 to 1306 in 28b2e76

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1204 to 1206 in 28b2e76

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L494-L496
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L175-L177
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L122-L124
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L843-L845
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L755-L757
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L441-L443
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L667-L669
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L707-L709
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L293-L295
https://github.com/cockroachdb/cockroach/blob/28b2e767986169df857f5f2cd1790d71ac633395/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L525-L527

cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go

Lines 132 to 134 in 28b2e76

if was, is := args.Txn.WriteTimestamp, reply.RecoveredTxn.WriteTimestamp; was != is {
return result.Result{}, errors.AssertionFailedf(
"programming error: timestamp change by implicitly committed transaction: %v->%v", was, is, 

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 901
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2104
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2289
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 2287
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 247
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3086
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1036
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1709
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2155
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1648
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1645
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1913
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 324
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 286
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 254
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 250
pkg/sql/colexec/limit.go in pkg/sql/colexec.(*limitOp).Next at line 47
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 124
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 427
bazel-out/k8-opt/bin/pkg/sql/colexec/ordered_synchronizer.eg.go in pkg/sql/colexec.(*OrderedSynchronizer).Next at line 119
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 383
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1006
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 900
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1279
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1708
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2182
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 210
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1044
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 921
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1036
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 227
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 851
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 241
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 243
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 276
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 96
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 841
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 551
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 779
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1044
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 921
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1036
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 227
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 851
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 241
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 243
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 276
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 96
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 841
pkg/server/node.go in pkg/server.(*Node).Batch at line 1305
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1205
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 176
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 123
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 844
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 756
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 442
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 668
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 708
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 294
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 526
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 133

Tags

Tag Value
Command server
Environment v23.1.9
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.9
Cockroach SHA 28b2e76
# of CPUs 6
# of Goroutines 776

Jira issue: CRDB-35685
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Jan 26, 2024
@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1706233945.398780464,2->1706233938.623665726,0 (1) attach... Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1706233945.398780464,2->1706233938.623665726,0 (1) attach... Mar 15, 2024
@yuzefovich
Copy link
Member

fixed by #107658

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry