Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... #94907

Closed
cockroach-teamcity opened this issue Jan 9, 2023 · 1 comment
Closed

sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... #94907

cockroach-teamcity opened this issue Jan 9, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 9, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3862674888/?referrer=webhooks_plugin

Panic message:

validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
(1) keys: [sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation]
Wraps: (2) secondary error attachment
| relation × (69): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
| (1) secondary error attachment
| | relation × (70): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
| | (1) secondary error attachment
| | | relation × (71): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
| | | (1) secondary error attachment
| | | | relation × (72): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
| | | | (1) keys: [sql.schema.validation_errors.read.cross_references.relation]
| | | | Wraps: (2) attached stack trace
| | | | -- stack trace:
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| | | | | [...repeated from below...]
| | | | Wraps: (3) relation × (72)
| | | | Wraps: (4) assertion failure
| | | | Wraps: (5) attached stack trace
| | | | -- stack trace:
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateInboundTableRef
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:298
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:188
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree.Catalog.Validate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree/catalog.go:144
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:279
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetAllDescriptors
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:284
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupresolver.LoadAllDescs.func1
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupresolver/targets.go:637
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:94
| | | | | github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| | | | | github.com/cockroachdb/cockroach/pkg/kv/db.go:949
| | | | | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| | | | | github.com/cockroachdb/cockroach/pkg/kv/txn.go:993
| | | | | github.com/cockroachdb/cockroach/pkg/kv.runTxn
| | | | | github.com/cockroachdb/cockroach/pkg/kv/db.go:948
| | | | | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| | | | | github.com/cockroachdb/cockroach/pkg/kv/db.go:911
| | | | | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| | | | | github.com/cockroachdb/cockroach/pkg/kv/db.go:890
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:80
| | | | | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
| | | | | github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3383
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupresolver.LoadAllDescs
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupresolver/targets.go:632
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.fullClusterTargetsBackup
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/pkg/ccl/backupccl/targets.go:310
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.backupPlanHook.func3
| | | | | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/pkg/ccl/backupccl/backup_planning.go:800
| | | | | github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1
| | | | | github.com/cockroachdb/cockroach/pkg/sql/planhook.go:178
| | | | | runtime.goexit
| | | | | GOROOT/src/runtime/asm_amd64.s:1581
| | | | Wraps: (6) invalid depended-on-by relation back reference
| | | | Wraps: (7) referenced table ID 66: referenced descriptor not found
| | | | | -- cause hidden behind barrier
| | | | | referenced table ID 66: referenced descriptor not found
| | | | | (1) attached stack trace
| | | | | -- stack trace:
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog.WrapTableDescRefErr
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/errors.go:164
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationDescGetterImpl).GetTableDescriptor
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:284
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateInboundTableRef
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:296
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:188
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree.Catalog.Validate
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree/catalog.go:144
| | | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors
| ...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 213 to 215 in 3b76f78

case catalog.Table:
err = errors.Wrapf(err, catalog.Table+" %q (%d)", name, id)
case catalog.Database:
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 148 to 150 in 3b76f78

}
vea.errors = append(vea.errors, vea.decorate(err))
}
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report

cockroach/pkg/sql/catalog/tabledesc/validate.go

Lines 187 to 189 in 3b76f78

for _, by := range desc.DependedOnBy {
vea.Report(desc.validateInboundTableRef(by, vdg))
}
in pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 76 to 78 in 3b76f78

if !desc.Dropped() {
desc.ValidateCrossReferences(&vea, vdg)
}
in pkg/sql/catalog/internal/validate.Validate.func2

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 164 to 166 in 3b76f78

vea.currentDescriptor = desc
validationFn(desc)
}
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 71 to 73 in 3b76f78

// Descriptor cross-reference checks.
if !vea.validateDescriptorsAtLevel(
catalog.ValidationLevelCrossReferences,
in pkg/sql/catalog/internal/validate.Validate

cockroach/pkg/sql/catalog/nstree/catalog.go

Lines 143 to 145 in 3b76f78

) (ve catalog.ValidationErrors) {
return validate.Validate(ctx, version, c, telemetry, targetLevel, descriptors...)
}
in pkg/sql/catalog/nstree.Catalog.Validate

cockroach/pkg/sql/catalog/descs/kv_descriptors.go

Lines 278 to 280 in 3b76f78

descs := c.OrderedDescriptors()
ve := c.Validate(ctx, version, catalog.ValidationReadTelemetry, catalog.ValidationLevelCrossReferences, descs...)
if err := ve.CombinedError(); err != nil {
in pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors

cockroach/pkg/sql/catalog/descs/collection.go

Lines 283 to 285 in 3b76f78

func (tc *Collection) GetAllDescriptors(ctx context.Context, txn *kv.Txn) (nstree.Catalog, error) {
return tc.kv.getAllDescriptors(ctx, txn, tc.version)
}
in pkg/sql/catalog/descs.(*Collection).GetAllDescriptors

cockroach/pkg/ccl/backupccl/backupresolver/targets.go

Lines 636 to 638 in 3b76f78

}
all, err := col.GetAllDescriptors(ctx, txn)
allDescs = all.OrderedDescriptors()
in pkg/ccl/backupccl/backupresolver.LoadAllDescs.func1

cockroach/pkg/sql/catalog/descs/txn.go

Lines 93 to 95 in 3b76f78

}
if err := f(ctx, txn, &descsCol); err != nil {
return err
in pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2

cockroach/pkg/kv/db.go

Lines 948 to 950 in 3b76f78

err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error {
return retryable(ctx, txn)
})
in pkg/kv.runTxn.func1

cockroach/pkg/kv/txn.go

Lines 992 to 994 in 3b76f78

}
err = fn(ctx, txn)
in pkg/kv.(*Txn).exec

cockroach/pkg/kv/db.go

Lines 947 to 949 in 3b76f78

func runTxn(ctx context.Context, txn *Txn, retryable func(context.Context, *Txn) error) error {
err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error {
return retryable(ctx, txn)
in pkg/kv.runTxn

cockroach/pkg/kv/db.go

Lines 910 to 912 in 3b76f78

txn.SetDebugName("unnamed")
return runTxn(ctx, txn, retryable)
}
in pkg/kv.(*DB).TxnWithAdmissionControl

cockroach/pkg/kv/db.go

Lines 889 to 891 in 3b76f78

func (db *DB) Txn(ctx context.Context, retryable func(context.Context, *Txn) error) error {
return db.TxnWithAdmissionControl(
ctx, roachpb.AdmissionHeader_OTHER, admission.NormalPri, retryable)
in pkg/kv.(*DB).Txn

cockroach/pkg/sql/catalog/descs/txn.go

Lines 79 to 81 in 3b76f78

var descsCol Collection
if err := db.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
modifiedDescriptors = nil
in pkg/sql/catalog/descs.(*CollectionFactory).Txn

cockroach/pkg/sql/exec_util.go

Lines 3382 to 3384 in 3b76f78

) error {
return execCfg.CollectionFactory.Txn(ctx, execCfg.InternalExecutor, execCfg.DB, f)
}
in pkg/sql.DescsTxn

cockroach/pkg/ccl/backupccl/backupresolver/targets.go

Lines 631 to 633 in 3b76f78

) (allDescs []catalog.Descriptor, _ error) {
if err := sql.DescsTxn(ctx, execCfg, func(ctx context.Context, txn *kv.Txn, col *descs.Collection) error {
err := txn.SetFixedTimestamp(ctx, asOf)
in pkg/ccl/backupccl/backupresolver.LoadAllDescs
https://github.com/cockroachdb/cockroach/blob/3b76f78d724dfc1e7bc8d697f5a7de960d8d1e98/pkg/ccl/backupccl/pkg/ccl/backupccl/targets.go#L309-L311 in pkg/ccl/backupccl.fullClusterTargetsBackup
https://github.com/cockroachdb/cockroach/blob/3b76f78d724dfc1e7bc8d697f5a7de960d8d1e98/pkg/ccl/backupccl/pkg/ccl/backupccl/backup_planning.go#L799-L801 in pkg/ccl/backupccl.backupPlanHook.func3

cockroach/pkg/sql/planhook.go

Lines 177 to 179 in 3b76f78

defer sp.Finish()
err := f.f(subplanCtx, f.subplans, f.run.resultsCh)
select {
in pkg/sql.(*hookFnNode).startExec.func1
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate at line 214
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report at line 149
pkg/sql/catalog/tabledesc/validate.go in pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences at line 188
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate.func2 at line 77
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel at line 165
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate at line 72
pkg/sql/catalog/nstree/catalog.go in pkg/sql/catalog/nstree.Catalog.Validate at line 144
pkg/sql/catalog/descs/kv_descriptors.go in pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors at line 279
pkg/sql/catalog/descs/collection.go in pkg/sql/catalog/descs.(*Collection).GetAllDescriptors at line 284
pkg/ccl/backupccl/backupresolver/targets.go in pkg/ccl/backupccl/backupresolver.LoadAllDescs.func1 at line 637
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).Txn.func2 at line 94
pkg/kv/db.go in pkg/kv.runTxn.func1 at line 949
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 993
pkg/kv/db.go in pkg/kv.runTxn at line 948
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 911
pkg/kv/db.go in pkg/kv.(*DB).Txn at line 890
pkg/sql/catalog/descs/txn.go in pkg/sql/catalog/descs.(*CollectionFactory).Txn at line 80
pkg/sql/exec_util.go in pkg/sql.DescsTxn at line 3383
pkg/ccl/backupccl/backupresolver/targets.go in pkg/ccl/backupccl/backupresolver.LoadAllDescs at line 632
pkg/ccl/backupccl/pkg/ccl/backupccl/targets.go in pkg/ccl/backupccl.fullClusterTargetsBackup at line 310
pkg/ccl/backupccl/pkg/ccl/backupccl/backup_planning.go in pkg/ccl/backupccl.backupPlanHook.func3 at line 800
pkg/sql/planhook.go in pkg/sql.(*hookFnNode).startExec.func1 at line 178
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581
Tag Value
Cockroach Release v22.1.12
Cockroach SHA: 3b76f78
Platform linux amd64
Distribution CCL
Environment v22.1.12
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-23220
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jan 9, 2023
@yuzefovich
Copy link
Member

dup of #91645

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity