Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v22.1.0: sql: alter table attempted to update job for mutation 2, but job already exists with mutation 1 #82046

Closed
cockroach-teamcity opened this issue May 29, 2022 · 3 comments
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 29, 2022

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3310675850/?referrer=webhooks_plugin

Panic message:

table.go:202: attempted to update job for mutation 2, but job already exists with mutation 1
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).createOrUpdateSchemaChangeJob
| github.com/cockroachdb/cockroach/pkg/sql/table.go:202
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeSchemaChange
| github.com/cockroachdb/cockroach/pkg/sql/table.go:248
| github.com/cockroachdb/cockroach/pkg/sql.(*alterTableNode).startExec
| github.com/cockroachdb/cockroach/pkg/sql/alter_table.go:852
| github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:516
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
| github.com/cockroachdb/cockroach/pkg/sql.walkPlan
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
| github.com/cockroachdb/cockroach/pkg/sql.startExec
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:519
| github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
| github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:146
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:158
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1
| github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:248
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start
| github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:120
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:119
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:732
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:472
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:246
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:581
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1429
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1467
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1143
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:686
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:143
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:231
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1951
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1953
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1799
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:747
Wraps: (4) attempted to update job for mutation 2, but job already exists with mutation 1
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: attempted to update job for mutation 2, but job already exists with mutation 1
table.go:202: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/table.go

Lines 201 to 203 in 5b78463

if mutationID != descpb.InvalidMutationID && mutationID != oldDetails.TableMutationID {
return errors.AssertionFailedf(
"attempted to update job for mutation %d, but job already exists with mutation %d",
in pkg/sql.(*planner).createOrUpdateSchemaChangeJob

cockroach/pkg/sql/table.go

Lines 247 to 249 in 5b78463

if !tableDesc.IsNew() {
if err := p.createOrUpdateSchemaChangeJob(ctx, tableDesc, jobDesc, mutationID); err != nil {
return err
in pkg/sql.(*planner).writeSchemaChange
}
if err := params.p.writeSchemaChange(
params.ctx, n.tableDesc, mutationID, tree.AsStringWithFQNames(n.n, params.Ann()),
in pkg/sql.(*alterTableNode).startExec

cockroach/pkg/sql/plan.go

Lines 515 to 517 in 5b78463

}
return n.startExec(params)
},
in pkg/sql.startExec.func2

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 5b78463

}
v.err = v.observer.leaveNode(name, plan)
}()
in pkg/sql.(*planVisitor).visitInternal.func1

cockroach/pkg/sql/walk.go

Lines 296 to 298 in 5b78463

}
}
in pkg/sql.(*planVisitor).visitInternal
}
v.visitInternal(plan, name)
return plan
in pkg/sql.(*planVisitor).visit
v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err
in pkg/sql.walkPlan

cockroach/pkg/sql/plan.go

Lines 518 to 520 in 5b78463

}
return walkPlan(params.ctx, plan, o)
}
in pkg/sql.startExec
// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)
in pkg/sql.(*planNodeToRowSource).Start
ctx = c.StartInternalNoSpan(ctx)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {
in pkg/sql/colexec.(*Columnarizer).Init
func (bic *batchInfoCollector) Init(ctx context.Context) {
bic.Operator.Init(ctx)
bic.mu.Lock()
in pkg/sql/colflow.(*batchInfoCollector).Init
if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
}); err != nil {
in pkg/sql/colexec.(*Materializer).Start.func1
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
// reached), so we need to wrap it with a catcher.
if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
in pkg/sql/colexec.(*Materializer).Start
if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
}); err != nil {
in pkg/sql/colflow.(*FlowCoordinator).Start.func1
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
ctx = f.StartInternalNoSpan(ctx)
if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
in pkg/sql/colflow.(*FlowCoordinator).Start
}
pb.self.Start(ctx)
Run(pb.Ctx, pb.self, pb.Output)
in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc)
headProc.Run(ctx)
}
in pkg/sql/flowinfra.(*FlowBase).Run
// as the root, so we run this flow with the default implementation.
f.FlowBase.Run(ctx, doneFn)
return
in pkg/sql/colflow.(*vectorizedFlow).Run
// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run
recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun
// the planner whether or not to plan remote table readers.
cleanup := ex.server.cfg.DistSQLPlanner.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*connExecutor).execWithDistSQLEngine
ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err := ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine
if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState
} else {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
}
in pkg/sql.(*connExecutor).execStmt
}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal
canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execCmd.func2
return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd
var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run
}()
return h.ex.run(ctx, s.pool, reserved, cancel)
}
in pkg/sql.(*Server).ServeConn

pkg/sql/table.go in pkg/sql.(*planner).createOrUpdateSchemaChangeJob at line 202
pkg/sql/table.go in pkg/sql.(*planner).writeSchemaChange at line 248
pkg/sql/alter_table.go in pkg/sql.(*alterTableNode).startExec at line 852
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 516
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/plan.go in pkg/sql.startExec at line 519
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 146
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 158
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 90
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start.func1 at line 248
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start at line 247
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start.func1 at line 120
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start at line 119
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 732
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 472
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 246
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 581
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1429
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1467
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1143
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 686
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 143
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 231
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1951
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1953
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1799
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 747
Tag Value
Cockroach Release v22.1.0
Cockroach SHA: 5b78463
Platform linux amd64
Distribution CCL
Environment v22.1.0
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-16185

@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels May 29, 2022
@rafiss rafiss changed the title sentry: table.go:202: attempted to update job for mutation 2, but job already exists with mutation 1 (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: | github.com/cockroach... v22.1.0: sql: alter table attempted to update job for mutation 2, but job already exists with mutation 1 May 29, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label May 29, 2022
@ajwerner
Copy link
Contributor

ajwerner commented Jun 7, 2022

Let's wait and see on this.

@postamar
Copy link
Contributor

postamar commented Jun 7, 2022

This seems unactionable for the time being, let's wait and see.

@postamar
Copy link
Contributor

Won't fix. We're assuming somebody did something invalid intentionally until proven otherwise.

@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Development

No branches or pull requests

3 participants