Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvcoord: v23.1.20: transaction unexpectedly committed #123976

Closed
cockroach-sentry opened this issue May 11, 2024 · 2 comments
Closed

kvcoord: v23.1.20: transaction unexpectedly committed #123976

cockroach-sentry opened this issue May 11, 2024 · 2 comments
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-kv KV Team

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented May 11, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5333151409/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:1013: transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14: TransactionStatusError: already committed (REASON_TXN_COMMITTED)
(1) assertion failure
Wraps: (2) issue: https://github.com/cockroachdb/cockroach/issues/103817
  | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support.
Wraps: (3)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1013
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:977
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:537
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1036
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1060
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:685
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:700
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1450
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransaction
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1332
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:854
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3140
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2194
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2199
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2116
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:911
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1172
Wraps: (4) transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14
Wraps: (5) TransactionStatusError: already committed (REASON_TXN_COMMITTED)
Error types: (1) *assert.withAssertionFailure (2) *issuelink.withIssueLink (3) *errbase.opaqueWrapper (4) *errutil.withPrefix (5) *kvpb.TransactionStatusError
-- report composition:
*kvpb.TransactionStatusError
*errutil.withPrefix: transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14
txn_coord_sender.go:1013: *withstack.withStack (top exception)
*issuelink.withIssueLink: https://github.com/cockroachdb/cockroach/issues/103817
*assert.withAssertionFailure
Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s#L1171-L1173

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 93a6798

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(
ctx,

cockroach/pkg/sql/conn_executor.go

Lines 910 to 912 in 93a6798

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}

cockroach/pkg/sql/conn_executor.go

Lines 2115 to 2117 in 93a6798

var err error
if err = ex.execCmd(); err != nil {
// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2198 to 2200 in 93a6798

return err
}()
// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2193 to 2195 in 93a6798

tcmd.LastInBatch || !implicitTxnForBatch)
ev, payload, err = ex.execStmt(
ctx, tcmd.Statement, nil /* portal */, nil /* pinfo */, stmtRes, canAutoCommit,

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 93a6798

}
err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3139 to 3141 in 93a6798

} else {
err = op(ctx)
}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 93a6798

err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 853 to 855 in 93a6798

// CommitTransaction is executed fully here; there's no plan for it.
ev, payload := ex.commitSQLTransaction(ctx, ast, ex.commitSQLTransactionInternal)
return ev, payload, nil

cockroach/pkg/sql/conn_executor_exec.go

Lines 1331 to 1333 in 93a6798

ex.phaseTimes.SetSessionPhaseTime(sessionphase.SessionStartTransactionCommit, timeutil.Now())
if err := commitFn(ctx); err != nil {
// For certain retryable errors, we should turn them into client visible

cockroach/pkg/sql/conn_executor_exec.go

Lines 1449 to 1451 in 93a6798

if err := ex.state.mu.txn.Commit(ctx); err != nil {
return err

cockroach/pkg/kv/txn.go

Lines 699 to 701 in 93a6798

return txn.commit(ctx)
}

cockroach/pkg/kv/txn.go

Lines 684 to 686 in 93a6798

ba := &kvpb.BatchRequest{Requests: et.unionArr[:]}
_, pErr := txn.Send(ctx, ba)
if pErr == nil {

cockroach/pkg/kv/txn.go

Lines 1059 to 1061 in 93a6798

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1035 to 1037 in 93a6798

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 536 to 538 in 93a6798

pErr = tc.updateStateLocked(ctx, ba, br, pErr)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 976 to 978 in 93a6798

if errTxn := pErr.GetTxn(); errTxn != nil {
if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
return kvpb.NewError(err)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 1012 to 1014 in 93a6798

// are committed.
err := errors.Wrapf(pErrWithTxn.GoError(),
"transaction unexpectedly committed, ba: %s. txn: %s", 

src/runtime/asm_arm64.s in runtime.goexit at line 1172
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 911
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2116
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2199
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2194
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3140
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 854
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).commitSQLTransaction at line 1332
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).commitSQLTransactionInternal at line 1450
pkg/kv/txn.go in pkg/kv.(*Txn).Commit at line 700
pkg/kv/txn.go in pkg/kv.(*Txn).commit at line 685
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1060
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1036
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 537
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 977
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 1013

Tags

Tag Value
Command server
Environment v23.1.20
Go Version go1.19.13
Platform linux arm64
Distribution CCL
Cockroach Release v23.1.20
Cockroach SHA 93a6798
# of CPUs 32
# of Goroutines 14669

Jira issue: CRDB-38634
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels May 11, 2024
@yuzefovich
Copy link
Member

I'm assuming this was supposed to be fixed via #103817, but apparently not.

@yuzefovich yuzefovich changed the title Sentry: txn_coord_sender.go:1013: transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.009... kvcoord: v23.1.20: transaction unexpectedly committed May 23, 2024
@yuzefovich yuzefovich added the T-kv KV Team label May 23, 2024
@yuzefovich yuzefovich mentioned this issue May 23, 2024
Closed
@arulajmani
Copy link
Collaborator

#107658 didn't make it to 23.1, so I'll close this out.

@github-project-automation github-project-automation bot moved this to Incoming in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-kv KV Team
Projects
No open projects
KV
Status: Incoming
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yuzefovich @arulajmani @cockroach-sentry