Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: txn_coord_sender.go:976: log.Fatal: transaction unexpectedly committed, ba: Put [/Table/181/1/×/×,/Min), Delete [/Table/181/4/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/181/4/×/×/�... #108686

Closed
cockroach-sentry opened this issue Aug 13, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Aug 13, 2023

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4392883017/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:976: log.Fatal: transaction unexpectedly committed, ba: Put [/Table/181/1/×/×,/Min), Delete [/Table/181/4/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/181/4/×/×/×/×/×/×/×,/Min), Delete [/Table/181/8/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/181/8/×/×/×/×/×/×/×,/Min), EndTxn(commit) [/Table/181/10/×/×], [txn: 0cfa78af]. txn: "sql txn" meta={id=0cfa78af key=/Table/181/10/×/× pri=0.03677262 epo=0 ts=1691966764.780172886,1 min=1691966754.260724675,0 seq=6} lock=true stat=COMMITTED rts=1691966764.780172886,1 wto=false gul=1691966754.760724675,0 int=6 ifw=5: ×
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:976
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:925
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:528
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1016
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1093
  | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:855
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:673
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:741
  | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:228
  | github.com/cockroachdb/cockroach/pkg/sql.(*updateNode).BatchedNext
  | 	github.com/cockroachdb/cockroach/pkg/sql/update.go:197
  | github.com/cockroachdb/cockroach/pkg/sql.(*serializeNode).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:115
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:211
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:243
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.constNullOp.Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go:611
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*castOpNullAny).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go:1160
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:252
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:256
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:299
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:757
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1764
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1472
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1597
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1200
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:694
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2438
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130
Wraps: (2) secondary error attachment
  | transaction unexpectedly committed, ba: Put [/Table/181/1/×/×,/Min), Delete [/Table/181/4/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/181/4/×/×/×/×/×/×/×,/Min), Delete [/Table/181/8/×/×/×/×/×/×/×,/Min), ConditionalPut [/Table/181/8/×/×/×/×/×/×/×,/Min), EndTxn(commit) [/Table/181/10/×/×], [txn: 0cfa78af]. txn: "sql txn" meta={id=0cfa78af key=/Table/181/10/×/× pri=0.03677262 epo=0 ts=1691966764.780172886,1 min=1691966754.260724675,0 seq=6} lock=true stat=COMMITTED rts=1691966764.780172886,1 wto=false gul=1691966754.760724675,0 int=6 ifw=5: ×
  | (1) assertion failure
  | Wraps: (2) issue: https://github.com/cockroachdb/cockroach/issues/67765
  |   | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support. This assertion can be disabled by setting the environment variable COCKROACH_DISABLE_COMMIT_SANITY_CHECK=true
  | Wraps: (3) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:963
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:925
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:528
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1016
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1093
  |   | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:855
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:673
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:741
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:228
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*updateNode).BatchedNext
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/update.go:197
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*serializeNode).Next
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:115
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_so...
Stacktrace (expand for inline code snippets):

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)

} else {
err = op(ctx)
}

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

}
err := ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
return recv.stats, err

defer setupCleanup()
cleanup := dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
return func() {

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}

for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {

func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}

}()
operation()
return retErr

func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}

func (d *simpleProjectOp) Next() coldata.Batch {
batch := d.Input.Next()
if batch.Length() == 0 {

https://github.com/cockroachdb/cockroach/blob/1b11ca676513ae073ea696c7cc3123d2684bc4bd/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L1159-L1161
func (e *vectorTypeEnforcer) Next() coldata.Batch {
b := e.Input.Next()
if b.Length() == 0 {

https://github.com/cockroachdb/cockroach/blob/1b11ca676513ae073ea696c7cc3123d2684bc4bd/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L610-L612
func (e *vectorTypeEnforcer) Next() coldata.Batch {
b := e.Input.Next()
if b.Length() == 0 {

for batchDone := false; !batchDone; {
row, meta := c.input.Next()
if meta != nil {

for p.State == execinfra.StateRunning {
valid, err := p.node.Next(p.params)
if err != nil || !valid {

// First batch, or finished previous batch; advance one.
if next, err := s.source.BatchedNext(params); !next {
return false, err

cockroach/pkg/sql/update.go

Lines 196 to 198 in 1b11ca6

u.run.tu.setRowsWrittenLimit(params.extendedEvalCtx.SessionData())
if err := u.run.tu.finalize(params.ctx); err != nil {
return false, err

// coordinator.
err = tb.txn.CommitInBatch(ctx, tb.b)
} else {

cockroach/pkg/kv/txn.go

Lines 740 to 742 in 1b11ca6

b.initResult(1 /* calls */, 0, b.raw, nil)
return txn.Run(ctx, b)
}

cockroach/pkg/kv/txn.go

Lines 672 to 674 in 1b11ca6

}
return sendAndFill(ctx, txn.Send, b)
}

cockroach/pkg/kv/db.go

Lines 854 to 856 in 1b11ca6

ba.AdmissionHeader = b.AdmissionHeader
b.response, b.pErr = send(ctx, ba)
b.fillResults(ctx)

cockroach/pkg/kv/txn.go

Lines 1092 to 1094 in 1b11ca6

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1015 to 1017 in 1b11ca6

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

pErr = tc.updateStateLocked(ctx, ba, br, pErr)

if errTxn := pErr.GetTxn(); errTxn != nil {
if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
return roachpb.NewError(err)

if !DisableCommitSanityCheck && !knobs.DisableCommitSanityCheck {
log.Fatalf(ctx, "%s", err)
}

pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 130
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2438
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 131
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 694
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1200
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1597
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1472
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1764
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 757
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 320
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 299
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 256
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 252
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 124
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go in pkg/sql/colexec/colexecbase.(*castOpNullAny).Next at line 1160
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 135
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go in pkg/sql/colexec/colexecbase.constNullOp.Next at line 611
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 135
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 243
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Next at line 211
pkg/sql/plan_batch.go in pkg/sql.(*serializeNode).Next at line 115
pkg/sql/update.go in pkg/sql.(*updateNode).BatchedNext at line 197
pkg/sql/tablewriter.go in pkg/sql.(*tableWriterBase).finalize at line 228
pkg/kv/txn.go in pkg/kv.(*Txn).CommitInBatch at line 741
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 673
pkg/kv/db.go in pkg/kv.sendAndFill at line 855
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1093
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1016
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 528
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 925
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 976

Tags

Tag Value
Command server
Environment v22.2.9
Go Version go1.19.6
Platform linux amd64
Distribution CCL
Cockroach Release v22.2.9
Cockroach SHA 1b11ca6
# of CPUs 64
# of Goroutines 11593

Jira issue: CRDB-30590

@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Aug 13, 2023
@yuzefovich
Copy link
Member

dup of #67765

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

2 participants