Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... #94918

Closed
cockroach-teamcity opened this issue Jan 9, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 9, 2023

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3863264251/?referrer=webhooks_plugin

Panic message:

validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found
(1) keys: [sql.schema.validation_errors.read.cross_references.relation]
Wraps: (2) keys: [sql.schema.validation_errors.read.cross_references.relation]
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:188
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.lookupDescriptorsAndValidate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/descriptors.go:71
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.MustGetDescriptorsByID
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/descriptors.go:164
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getByIDs
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:253
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:200
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getByName.func1
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:355
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).withReadFromStore
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:374
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:344
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByNameIgnoringRequiredAndType
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:200
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:67
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetObjectDesc
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:41
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject
| github.com/cockroachdb/cockroach/pkg/sql/resolver.go:200
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExisting
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:410
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveExistingObject
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:193
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver.ResolveMutableExistingTableObject
| github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver/resolver.go:152
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).ResolveMutableTableDescriptor
| github.com/cockroachdb/cockroach/pkg/sql/resolver.go:76
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepareDrop
| github.com/cockroachdb/cockroach/pkg/sql/drop_table.go:164
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).DropSequence
| github.com/cockroachdb/cockroach/pkg/sql/drop_sequence.go:48
| github.com/cockroachdb/cockroach/pkg/sql.planOpaque
| github.com/cockroachdb/cockroach/pkg/sql/opaque.go:173
| github.com/cockroachdb/cockroach/pkg/sql.buildOpaque
| github.com/cockroachdb/cockroach/pkg/sql/opaque.go:68
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/opaque.go:69
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:377
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:241
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build
| github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:215
| github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
| github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:553
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
| github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:226
Wraps: (4) relation × (67)
Wraps: (5) assertion failure
Wraps: (6) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateInboundTableRef
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:298
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:188
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.lookupDescriptorsAndValidate
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/descriptors.go:71
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.MustGetDescriptorsByID
| github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/descriptors.go:164
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getByIDs
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:253
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:200
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getByName.func1
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:355
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).withReadFromStore
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:374
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:344
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByNameIgnoringRequiredAndType
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:200
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getObjectByName
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:67
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetObjectDesc
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/object.go:41
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject
| github.com/cockroachdb/cockroach/pkg/sql/res...

Stacktrace (expand for inline code snippets):

case catalog.Table:
err = errors.Wrapf(err, catalog.Table+" %q (%d)", name, id)
case catalog.Database:
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
}
vea.errors = append(vea.errors, vea.decorate(err))
}
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
for _, by := range desc.DependedOnBy {
vea.Report(desc.validateInboundTableRef(by, vdg))
}
in pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
if !desc.Dropped() {
desc.ValidateCrossReferences(&vea, vdg)
}
in pkg/sql/catalog/internal/validate.Validate.func2
vea.currentDescriptor = desc
validationFn(desc)
}
in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
// Descriptor cross-reference checks.
if !vea.validateDescriptorsAtLevel(
catalog.ValidationLevelCrossReferences,
in pkg/sql/catalog/internal/validate.Validate
}
ve := validate.Validate(ctx, version, vd, catalog.ValidationReadTelemetry, catalog.ValidationLevelCrossReferences, descs...)
if err := ve.CombinedError(); err != nil {
in pkg/sql/catalog/internal/catkv.lookupDescriptorsAndValidate
}
return lookupDescriptorsAndValidate(ctx, version, txn, cq, vd, ids)
}
in pkg/sql/catalog/internal/catkv.MustGetDescriptorsByID
}
kvDescs, err := catkv.MustGetDescriptorsByID(ctx, version, kd.codec, txn, vd, kvIDs, catalog.Any)
if err != nil {
in pkg/sql/catalog/descs.(*kvDescriptors).getByIDs
}
descs, err := kd.getByIDs(ctx, version, txn, vd, []descpb.ID{descID})
if err != nil {
in pkg/sql/catalog/descs.(*kvDescriptors).getByName
vd := tc.newValidationDereferencer(txn)
imm, err := tc.kv.getByName(ctx, version, txn, vd, uncommittedDB, parentID, parentSchemaID, name)
if err != nil {
in pkg/sql/catalog/descs.(*Collection).getByName.func1
) (descs []catalog.Descriptor, _ error) {
descs, err := readFn()
if err != nil {
in pkg/sql/catalog/descs.(*Collection).withReadFromStore
var descs []catalog.Descriptor
descs, err = tc.withReadFromStore(mutable, func() ([]catalog.Descriptor, error) {
// Try to re-use an unvalidated descriptor if there is one.
in pkg/sql/catalog/descs.(*Collection).getByName
prefix.Schema = sc
found, obj, err := tc.getByName(
ctx, txn, db, sc, objectName, flags.AvoidLeased, flags.RequireMutable, flags.AvoidSynthetic,
in pkg/sql/catalog/descs.(*Collection).getObjectByNameIgnoringRequiredAndType
const alwaysLookupLeasedPublicSchema = false
prefix, desc, err = tc.getObjectByNameIgnoringRequiredAndType(
ctx, txn, catalogName, schemaName, objectName, flags,
in pkg/sql/catalog/descs.(*Collection).getObjectByName
) (prefix catalog.ResolvedObjectPrefix, desc catalog.Descriptor, err error) {
return tc.getObjectByName(ctx, txn, db, schema, object, flags)
}
in pkg/sql/catalog/descs.(*Collection).GetObjectDesc
prefix, objMeta, err = sc.GetObjectDesc(ctx, p.txn, dbName, scName, obName, flags)
return objMeta != nil, prefix, objMeta, err
in pkg/sql.(*planner).LookupObject
if isVirtualSchema || curDb != "" {
if found, prefix, result, err = r.LookupObject(
ctx, lookupFlags, curDb, u.Schema(), u.Object(),
in pkg/sql/catalog/resolver.ResolveExisting
) (res catalog.Descriptor, _ catalog.ResolvedObjectPrefix, err error) {
found, prefix, obj, err := ResolveExisting(ctx, un, sc, lookupFlags, sc.CurrentDatabase(), sc.CurrentSearchPath())
if err != nil {
in pkg/sql/catalog/resolver.ResolveExistingObject
var desc catalog.Descriptor
desc, prefix, err = ResolveExistingObject(ctx, sc, un, lookupFlags)
if err != nil || desc == nil {
in pkg/sql/catalog/resolver.ResolveMutableExistingTableObject
) (prefix catalog.ResolvedObjectPrefix, table *tabledesc.Mutable, err error) {
prefix, desc, err := resolver.ResolveMutableExistingTableObject(ctx, p, tn, required, requiredType)
if err != nil {
in pkg/sql.(*planner).ResolveMutableTableDescriptor
) (*tabledesc.Mutable, error) {
_, tableDesc, err := p.ResolveMutableTableDescriptor(ctx, name, required, requiredType)
if err != nil {
in pkg/sql.(*planner).prepareDrop
tn := &n.Names[i]
droppedDesc, err := p.prepareDrop(ctx, tn, !n.IfExists, tree.ResolveRequireSequenceDesc)
if err != nil {
in pkg/sql.(*planner).DropSequence

cockroach/pkg/sql/opaque.go

Lines 172 to 174 in 3b76f78

case *tree.DropSequence:
return p.DropSequence(ctx, n)
case *tree.DropTable:
in pkg/sql.planOpaque
var err error
plan, err = planOpaque(ctx, p, stmt)
if err != nil {
in pkg/sql.buildOpaque
}
obj, err := info.buildFn(b.ctx, b.semaCtx, b.evalCtx, stmt)
if err != nil {
in pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque
// See if we have an opaque handler registered for this statement type.
if outScope := b.tryBuildOpaque(stmt, inScope); outScope != nil {
// The opaque handler may resolve objects; we don't care about caching
in pkg/sql/opt/optbuilder.(*Builder).buildStmt
b.ctes = nil
outScope = b.buildStmt(stmt, desiredTypes, inScope)
// Build With operators for any CTEs hoisted to the top level.
in pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot
// and physical properties.
outScope := b.buildStmtAtRoot(b.stmt, nil /* desiredTypes */)
in pkg/sql/opt/optbuilder.(*Builder).Build
bld := optbuilder.New(ctx, &p.semaCtx, p.EvalContext(), &opc.catalog, f, opc.p.stmt.AST)
if err := bld.Build(); err != nil {
return nil, err
in pkg/sql.(*optPlanningCtx).buildExecMemo
execMemo, err := opc.buildExecMemo(ctx)
if err != nil {
in pkg/sql.(*planner).makeOptimizerPlan
func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
if err := planner.makeOptimizerPlan(ctx); err != nil {
log.VEventf(ctx, 1, "optimizer plan failed: %v", err)
in pkg/sql.(*connExecutor).makeExecPlan

pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate at line 214
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report at line 149
pkg/sql/catalog/tabledesc/validate.go in pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences at line 188
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate.func2 at line 77
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel at line 165
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate at line 72
pkg/sql/catalog/internal/catkv/descriptors.go in pkg/sql/catalog/internal/catkv.lookupDescriptorsAndValidate at line 71
pkg/sql/catalog/internal/catkv/descriptors.go in pkg/sql/catalog/internal/catkv.MustGetDescriptorsByID at line 164
pkg/sql/catalog/descs/kv_descriptors.go in pkg/sql/catalog/descs.(*kvDescriptors).getByIDs at line 253
pkg/sql/catalog/descs/kv_descriptors.go in pkg/sql/catalog/descs.(*kvDescriptors).getByName at line 200
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getByName.func1 at line 355
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).withReadFromStore at line 374
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getByName at line 344
pkg/sql/catalog/descs/object.go in pkg/sql/catalog/descs.(*Collection).getObjectByNameIgnoringRequiredAndType at line 200
pkg/sql/catalog/descs/object.go in pkg/sql/catalog/descs.(*Collection).getObjectByName at line 67
pkg/sql/catalog/descs/object.go in pkg/sql/catalog/descs.(*Collection).GetObjectDesc at line 41
pkg/sql/resolver.go in pkg/sql.(*planner).LookupObject at line 200
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveExisting at line 410
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveExistingObject at line 193
pkg/sql/catalog/resolver/resolver.go in pkg/sql/catalog/resolver.ResolveMutableExistingTableObject at line 152
pkg/sql/resolver.go in pkg/sql.(*planner).ResolveMutableTableDescriptor at line 76
pkg/sql/drop_table.go in pkg/sql.(*planner).prepareDrop at line 164
pkg/sql/drop_sequence.go in pkg/sql.(*planner).DropSequence at line 48
pkg/sql/opaque.go in pkg/sql.planOpaque at line 173
pkg/sql/opaque.go in pkg/sql.buildOpaque at line 68
pkg/sql/opt/optbuilder/opaque.go in pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque at line 69
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmt at line 377
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot at line 241
pkg/sql/opt/optbuilder/builder.go in pkg/sql/opt/optbuilder.(*Builder).Build at line 215
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 553
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 226
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 1433
Tag Value
Cockroach Release v22.1.12
Cockroach SHA: 3b76f78
Platform linux amd64
Distribution CCL
Environment v22.1.12
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-23228

@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jan 9, 2023
@yuzefovich
Copy link
Member

dup of #91645

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... sentry: validate.go:298: relation × (67): invalid depended-on-by relation back reference: referenced table ID 66: referenced descriptor not found (1) keys: [sql.schema.validation_errors.read.cross_references.... Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

2 participants