Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... #93024

Closed
cockroach-teamcity opened this issue Dec 5, 2022 · 1 comment
Closed

sentry: validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... #93024

cockroach-teamcity opened this issue Dec 5, 2022 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 5, 2022 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3788262141/?referrer=webhooks_plugin

Panic message:

validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).handlePermanentSchemaChangeError
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:912
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.OnFailOrCancel
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2847
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func3
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1337
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1338
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1283
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:415
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:336
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (2) original error when rolling back mutations
Wraps: (3) keys: [sql.schema.validation_errors.write.cross_references.relation]
Wraps: (4) keys: [sql.schema.validation_errors.write.cross_references.relation]
Wraps: (5)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:237
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/validate.go:38
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).ValidateUncommittedDescriptors
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/validate.go:61
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:98
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:949
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:993
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:948
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:911
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:890
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:80
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txn
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2526
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).addConstraints
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:555
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:351
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1965
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:855
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2668
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2799
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1239
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1240
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:415
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:336
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) relation × (327)
Wraps: (7) assertion failure
Wraps: (8)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateOutboundFK
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:359
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:237
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/validate.go:38
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).ValidateUncommittedDescriptors
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/validate.go:61
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:98
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:949
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:993
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:948
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:911
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:890
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:80
| github.com/cockroachdb/cockroa...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/schema_changer.go

Lines 911 to 913 in 22fa382

// error, so we'll record the original error now.
secondary := errors.Wrap(err, "original error when rolling back mutations")
sqltelemetry.RecordError(ctx, secondary, &sc.settings.SV)
in pkg/sql.(*SchemaChanger).handlePermanentSchemaChangeError

cockroach/pkg/sql/schema_changer.go

Lines 2846 to 2848 in 22fa382

if rollbackErr := sc.handlePermanentSchemaChangeError(ctx, scErr, p.ExtendedEvalContext()); rollbackErr != nil {
switch {
in pkg/sql.schemaChangeResumer.OnFailOrCancel

cockroach/pkg/jobs/registry.go

Lines 1336 to 1338 in 22fa382

}()
err = resumer.OnFailOrCancel(onFailOrCancelCtx, execCtx, jobErr)
}()
in pkg/jobs.(*Registry).stepThroughStateMachine.func3

cockroach/pkg/jobs/registry.go

Lines 1337 to 1339 in 22fa382

err = resumer.OnFailOrCancel(onFailOrCancelCtx, execCtx, jobErr)
}()
if successOnFailOrCancel := err == nil; successOnFailOrCancel {
in pkg/jobs.(*Registry).stepThroughStateMachine

cockroach/pkg/jobs/registry.go

Lines 1282 to 1284 in 22fa382

}
return r.stepThroughStateMachine(ctx, execCtx, resumer, job, StatusReverting, err)
case StatusPauseRequested:
in pkg/jobs.(*Registry).stepThroughStateMachine

cockroach/pkg/jobs/adopt.go

Lines 414 to 416 in 22fa382

// Run the actual job.
err := r.stepThroughStateMachine(ctx, execCtx, resumer, job, status, finalResumeError)
// If the context has been canceled, disregard errors for the sake of logging
in pkg/jobs.(*Registry).runJob

cockroach/pkg/jobs/adopt.go

Lines 335 to 337 in 22fa382

// was one it's been set in the job status already.
_ = r.runJob(resumeCtx, resumer, job, status, job.taskName())
}); err != nil {
in pkg/jobs.(*Registry).resumeJob.func1

cockroach/pkg/util/stop/stopper.go

Lines 493 to 495 in 22fa382

f(ctx)
}()
in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/sql/schema_changer.go in pkg/sql.(*SchemaChanger).handlePermanentSchemaChangeError at line 912
pkg/sql/schema_changer.go in pkg/sql.schemaChangeResumer.OnFailOrCancel at line 2847
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func3 at line 1337
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1338
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1283
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 415
pkg/jobs/adopt.go in pkg/jobs.(*Registry).resumeJob.func1 at line 336
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 494
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581
Tag Value
Cockroach Release v22.1.11
Cockroach SHA: 22fa382
Platform linux amd64
Distribution CCL
Environment v22.1.11
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-22117
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Dec 5, 2022
@yuzefovich
Copy link
Member

dup of #91645

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... sentry: validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity