Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: txn_coord_sender.go:976: log.Fatal: transaction unexpectedly committed, ba: ConditionalPut [/Table/319/13/×/×/×/×,/Min), InitPut [/Table/319/17/×/×/×/×/×/×,/Min), InitPut [/Table/319... #109543

Closed
cockroach-sentry opened this issue Aug 25, 2023 · 1 comment
Closed

Sentry: txn_coord_sender.go:976: log.Fatal: transaction unexpectedly committed, ba: ConditionalPut [/Table/319/13/×/×/×/×,/Min), InitPut [/Table/319/17/×/×/×/×/×/×,/Min), InitPut [/Table/319... #109543

cockroach-sentry opened this issue Aug 25, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Aug 25, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4422130213/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:976: log.Fatal: transaction unexpectedly committed, ba: ConditionalPut [/Table/319/13/×/×/×/×,/Min), InitPut [/Table/319/17/×/×/×/×/×/×,/Min), InitPut [/Table/319/21/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/319/13/×/×/×/×], [txn: 786df38a]. txn: "sql txn" meta={id=786df38a key=/Table/319/13/×/×/×/× pri=0.00000000 epo=0 ts=1693006168.488578597,1 min=1693006167.602669554,0 seq=16} lock=true stat=COMMITTED rts=1693006168.488578597,1 wto=false gul=1693006172.602669554,0 int=15 ifw=14: ×
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:976
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:925
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:528
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1016
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1093
  | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:855
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:673
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:741
  | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:228
  | github.com/cockroachdb/cockroach/pkg/sql.(*insertFastPathNode).BatchedNext
  | 	github.com/cockroachdb/cockroach/pkg/sql/insert_fast_path.go:312
  | github.com/cockroachdb/cockroach/pkg/sql.(*rowCountNode).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:173
  | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  | github.com/cockroachdb/cockroach/pkg/sql.startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:521
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:166
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:122
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:121
  | github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:728
  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:480
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:307
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:757
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1764
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1472
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1599
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1202
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:694
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2440
Wraps: (2) secondary error attachment
  | transaction unexpectedly committed, ba: ConditionalPut [/Table/319/13/×/×/×/×,/Min), InitPut [/Table/319/17/×/×/×/×/×/×,/Min), InitPut [/Table/319/21/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/319/13/×/×/×/×], [txn: 786df38a]. txn: "sql txn" meta={id=786df38a key=/Table/319/13/×/×/×/× pri=0.00000000 epo=0 ts=1693006168.488578597,1 min=1693006167.602669554,0 seq=16} lock=true stat=COMMITTED rts=1693006168.488578597,1 wto=false gul=1693006172.602669554,0 int=15 ifw=14: ×
  | (1) assertion failure
  | Wraps: (2) issue: https://github.com/cockroachdb/cockroach/issues/67765
  |   | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support. This assertion can be disabled by setting the environment variable COCKROACH_DISABLE_COMMIT_SANITY_CHECK=true
  | Wraps: (3) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:963
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:925
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:528
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1016
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1093
  |   | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:855
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:673
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:741
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:228
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*insertFastPathNode).BatchedNext
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/insert_fast_path.go:312
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowCountNode).startExec
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:173
  |   | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  |   | github.com...
Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor_exec.go

Lines 2439 to 2441 in 15505fa

} else {
err = op(ctx)
}

cockroach/pkg/sql/conn_executor_exec.go

Lines 130 to 132 in 15505fa

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 693 to 695 in 15505fa

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1201 to 1203 in 15505fa

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 1598 to 1600 in 15505fa

}
err := ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
return recv.stats, err

cockroach/pkg/sql/distsql_running.go

Lines 1471 to 1473 in 15505fa

defer setupCleanup()
cleanup := dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1763 to 1765 in 15505fa

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
return func() {

cockroach/pkg/sql/distsql_running.go

Lines 756 to 758 in 15505fa

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 306 to 308 in 15505fa

// as the root, so we run this flow with the default implementation.
f.FlowBase.Run(ctx, doneFn)
return

cockroach/pkg/sql/flowinfra/flow.go

Lines 479 to 481 in 15505fa

log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc)
headProc.Run(ctx)
}

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 727 to 729 in 15505fa

}
pb.self.Start(ctx)
Run(pb.ctx, pb.self, pb.Output)

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 120 to 122 in 15505fa

ctx = f.StartInternal(ctx, "flow coordinator" /* name */)
if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 15505fa

}()
operation()
return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 121 to 123 in 15505fa

if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
}); err != nil {

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 165 to 167 in 15505fa

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)

cockroach/pkg/sql/plan.go

Lines 520 to 522 in 15505fa

}
return walkPlan(params.ctx, plan, o)
}

cockroach/pkg/sql/walk.go

Lines 42 to 44 in 15505fa

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err

cockroach/pkg/sql/walk.go

Lines 78 to 80 in 15505fa

}
v.visitInternal(plan, name)
return plan

cockroach/pkg/sql/walk.go

Lines 296 to 298 in 15505fa

}
}

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 15505fa

}
v.err = v.observer.leaveNode(name, plan)
}()

cockroach/pkg/sql/plan.go

Lines 517 to 519 in 15505fa

}
return n.startExec(params)
},

cockroach/pkg/sql/plan_batch.go

Lines 172 to 174 in 15505fa

for {
if next, err := r.source.BatchedNext(params); !next {
return err

cockroach/pkg/sql/insert_fast_path.go

Lines 311 to 313 in 15505fa

n.run.ti.setRowsWrittenLimit(params.extendedEvalCtx.SessionData())
if err := n.run.ti.finalize(params.ctx); err != nil {
return false, err

cockroach/pkg/sql/tablewriter.go

Lines 227 to 229 in 15505fa

// coordinator.
err = tb.txn.CommitInBatch(ctx, tb.b)
} else {

cockroach/pkg/kv/txn.go

Lines 740 to 742 in 15505fa

b.initResult(1 /* calls */, 0, b.raw, nil)
return txn.Run(ctx, b)
}

cockroach/pkg/kv/txn.go

Lines 672 to 674 in 15505fa

}
return sendAndFill(ctx, txn.Send, b)
}

cockroach/pkg/kv/db.go

Lines 854 to 856 in 15505fa

ba.AdmissionHeader = b.AdmissionHeader
b.response, b.pErr = send(ctx, ba)
b.fillResults(ctx)

cockroach/pkg/kv/txn.go

Lines 1092 to 1094 in 15505fa

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1015 to 1017 in 15505fa

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 527 to 529 in 15505fa

pErr = tc.updateStateLocked(ctx, ba, br, pErr)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 924 to 926 in 15505fa

if errTxn := pErr.GetTxn(); errTxn != nil {
if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
return roachpb.NewError(err)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 975 to 977 in 15505fa

if !DisableCommitSanityCheck && !knobs.DisableCommitSanityCheck {
log.Fatalf(ctx, "%s", err)
} 

pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2440
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 131
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 694
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1202
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1599
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1472
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1764
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 757
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 307
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 480
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 728
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start at line 121
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start.func1 at line 122
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 166
pkg/sql/plan.go in pkg/sql.startExec at line 521
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 518
pkg/sql/plan_batch.go in pkg/sql.(*rowCountNode).startExec at line 173
pkg/sql/insert_fast_path.go in pkg/sql.(*insertFastPathNode).BatchedNext at line 312
pkg/sql/tablewriter.go in pkg/sql.(*tableWriterBase).finalize at line 228
pkg/kv/txn.go in pkg/kv.(*Txn).CommitInBatch at line 741
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 673
pkg/kv/db.go in pkg/kv.sendAndFill at line 855
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1093
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1016
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 528
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 925
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 976

Tags

Tag Value
Command start-single-node
Environment v22.2.11
Go Version go1.19.6
Platform linux arm64
Distribution CCL
Cockroach Release v22.2.11
Cockroach SHA 15505fa
# of CPUs 6
# of Goroutines 369

Jira issue: CRDB-30975
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Aug 25, 2023
@yuzefovich
Copy link
Member

dup of #67765

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry