catalog: v22.1.9: referencing table was dropped without dependency unlinking #91645

cockroach-teamcity · 2022-11-10T06:22:46Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3731414904/?referrer=webhooks_plugin

Panic message:

type_desc.go:689: type × (184): referencing table 1045 was dropped without dependency unlinking
(1)
Wraps: (2) keys: [sql.schema.validation_errors.read.cross_references.type sql.schema.validation_errors.read.cross_references.type sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation sql.schema.validation_errors.read.cross_references.relation]
Wraps: (3) secondary error attachment
| type × (185): referencing table 1045 was dropped without dependency unlinking
| (1) secondary error attachment
| | relation × (190): origin table × (1045) is dropped
| | (1) secondary error attachment
| | | relation × (202): origin table × (1045) is dropped
| | | (1) secondary error attachment
| | | | relation × (206): origin table × (1045) is dropped
| | | | (1) keys: [sql.schema.validation_errors.read.cross_references.relation]
| | | | Wraps: (2) attached stack trace
| | | | -- stack trace:
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| | | | | [...repeated from below...]
| | | | Wraps: (3) relation × (206)
| | | | Wraps: (4) assertion failure
| | | | Wraps: (5) attached stack trace
| | | | -- stack trace:
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateInboundFK
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:385
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:240
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree.Catalog.Validate
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree/catalog.go:144
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:279
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetAllDescriptors
| | | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:284
| | | | | github.com/cockroachdb/cockroach/pkg/sql.glob..func75.1
| | | | | github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:426
| | | | | github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3
| | | | | github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:127
| | | | | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | | | | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| | | | | runtime.goexit
| | | | | GOROOT/src/runtime/asm_amd64.s:1581
| | | | Wraps: (6) origin table × (1045) is dropped
| | | | Error types: (1) *telemetrykeys.withTelemetry (2) *withstack.withStack (3) *errutil.withPrefix (4) *assert.withAssertionFailure (5) *withstack.withStack (6) *errutil.leafError
| | | Wraps: (2) keys: [sql.schema.validation_errors.read.cross_references.relation]
| | | Wraps: (3) attached stack trace
| | | -- stack trace:
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| | | | [...repeated from below...]
| | | Wraps: (4) relation × (202)
| | | Wraps: (5) assertion failure
| | | Wraps: (6) attached stack trace
| | | -- stack trace:
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).validateInboundFK
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:385
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).ValidateCrossReferences
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/validate.go:240
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate.func2
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:77
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:165
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.Validate
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:72
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree.Catalog.Validate
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree/catalog.go:144
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/kv_descriptors.go:279
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetAllDescriptors
| | | | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:284
| | | | github.com/cockroachdb/cockroach/pkg/sql.glob..func75.1
| | | | github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:426
| | | | github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3
| | | | github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:127
| | | | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | | | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| | | | runtime.goexit
| | | | GOROOT/src/runtime/asm_amd64.s:1581
| | | Wraps: (7) origin table × (1045) is dropped
| | | Error types: (1) *secondary.withSecondaryError (2) *telemetrykeys.withTelemetry (3) *withstack.withStack (4) *errutil.withPrefix (5) *assert.withAssertionFailure (6) *withstack.withStack (7) *errutil.leafError
| | Wraps: (2) keys: [sql.schema.validation_errors.read.cross_references.relation]
| | Wraps: (3) attached stack trace
| | -- stack trace:
| | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate
| | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:214
| | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report
| | | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate/validate.go:149
| | | [...repeated from below...]
| | Wraps: (4) relation × (190)
| | Wraps: (5) assertion failure
| | Wraps: (6) attached stack trace
| | -- stac...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 219 to 221 in e438c2f

    
           case catalog.Type: 
        
           	err = errors.Wrapf(err, catalog.Type+" %q (%d)", name, id) 
        
           default:

in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 148 to 150 in e438c2f

    
           	} 
        
           	vea.errors = append(vea.errors, vea.decorate(err)) 
        
           }

in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report

cockroach/pkg/sql/catalog/typedesc/type_desc.go

Lines 688 to 690 in e438c2f

    
           if tableDesc.Dropped() { 
        
           	vea.Report(errors.AssertionFailedf( 
        
           		"referencing table %d was dropped without dependency unlinking", id))

in pkg/sql/catalog/typedesc.(*immutable).ValidateCrossReferences

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 76 to 78 in e438c2f

    
           if !desc.Dropped() { 
        
           	desc.ValidateCrossReferences(&vea, vdg) 
        
           }

in pkg/sql/catalog/internal/validate.Validate.func2

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 164 to 166 in e438c2f

    
           	vea.currentDescriptor = desc 
        
           	validationFn(desc) 
        
           }

in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel

cockroach/pkg/sql/catalog/internal/validate/validate.go

Lines 71 to 73 in e438c2f

    
           // Descriptor cross-reference checks. 
        
           if !vea.validateDescriptorsAtLevel( 
        
           	catalog.ValidationLevelCrossReferences,

in pkg/sql/catalog/internal/validate.Validate

cockroach/pkg/sql/catalog/nstree/catalog.go

Lines 143 to 145 in e438c2f

    
           ) (ve catalog.ValidationErrors) { 
        
           	return validate.Validate(ctx, version, c, telemetry, targetLevel, descriptors...) 
        
           }

in pkg/sql/catalog/nstree.Catalog.Validate

cockroach/pkg/sql/catalog/descs/kv_descriptors.go

Lines 278 to 280 in e438c2f

    
           descs := c.OrderedDescriptors() 
        
           ve := c.Validate(ctx, version, catalog.ValidationReadTelemetry, catalog.ValidationLevelCrossReferences, descs...) 
        
           if err := ve.CombinedError(); err != nil {

in pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors

cockroach/pkg/sql/catalog/descs/collection.go

Lines 283 to 285 in e438c2f

    
           func (tc *Collection) GetAllDescriptors(ctx context.Context, txn *kv.Txn) (nstree.Catalog, error) { 
        
           	return tc.kv.getAllDescriptors(ctx, txn, tc.version) 
        
           }

in pkg/sql/catalog/descs.(*Collection).GetAllDescriptors

cockroach/pkg/sql/crdb_internal.go

Lines 425 to 427 in e438c2f

    
           worker := func(ctx context.Context, pusher rowPusher) error { 
        
           	all, err := p.Descriptors().GetAllDescriptors(ctx, p.txn) 
        
           	if err != nil {

in pkg/sql.glob..func75.1

cockroach/pkg/sql/virtual_table.go

Lines 126 to 128 in e438c2f

    
           } 
        
           err := worker(ctx, funcRowPusher(addRow)) 
        
           // If the query was canceled, next() will already return a

in pkg/sql.setupGenerator.func3

cockroach/pkg/util/stop/stopper.go

Lines 493 to 495 in e438c2f

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).decorate at line 220
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).Report at line 149
pkg/sql/catalog/typedesc/type_desc.go in pkg/sql/catalog/typedesc.(*immutable).ValidateCrossReferences at line 689
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate.func2 at line 77
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.(*validationErrorAccumulator).validateDescriptorsAtLevel at line 165
pkg/sql/catalog/internal/validate/validate.go in pkg/sql/catalog/internal/validate.Validate at line 72
pkg/sql/catalog/nstree/catalog.go in pkg/sql/catalog/nstree.Catalog.Validate at line 144
pkg/sql/catalog/descs/kv_descriptors.go in pkg/sql/catalog/descs.(*kvDescriptors).getAllDescriptors at line 279
pkg/sql/catalog/descs/collection.go in pkg/sql/catalog/descs.(*Collection).GetAllDescriptors at line 284
pkg/sql/crdb_internal.go in pkg/sql.glob..func75.1 at line 426
pkg/sql/virtual_table.go in pkg/sql.setupGenerator.func3 at line 127
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 494
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	`v22.1.9`
Cockroach SHA:	`e438c2f`
Platform	linux amd64
Distribution	CCL
Environment	v22.1.9
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-21361

The text was updated successfully, but these errors were encountered:

ajwerner · 2022-11-15T15:09:17Z

Seems like this is root caused by #82087, which is fixed but old clusters which encountered that bug will be plagued by these problems.

ajwerner · 2022-11-15T16:20:06Z

This is not related exactly to #82087, it's related to some cancellation of schema changes.

jordanlewis · 2022-11-16T04:27:53Z

This error message is easily reproducible with #91972 (though I suppose it's possible that this Sentry report was generated via different means).

postamar · 2022-11-22T15:09:08Z

#63161 will fix this

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Nov 10, 2022

blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Nov 10, 2022

ajwerner assigned fqazi Nov 15, 2022

ajwerner mentioned this issue Nov 16, 2022

sql: old schema changer doesn't handle type back references properly in alter column #91972

Open

ajwerner unassigned fqazi Nov 22, 2022

yuzefovich mentioned this issue Dec 13, 2022

sentry: validate.go:359: original error when rolling back mutations: relation × (327): referenced table × (326) is dropped (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... #93024

Closed

yuzefovich mentioned this issue Mar 9, 2023

sentry: validate.go:298: relation × (2514): invalid depended-on-by relation back reference: referenced table ID 2445: referenced descriptor not found (1) Wraps: (2) keys: [sql.schema.validation_errors.read.cr... #96171

Closed

exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023

rafiss closed this as completed Jun 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

catalog: v22.1.9: referencing table was dropped without dependency unlinking #91645

catalog: v22.1.9: referencing table was dropped without dependency unlinking #91645

cockroach-teamcity commented Nov 10, 2022 •

edited by cockroach-jira-scripts

Loading

ajwerner commented Nov 15, 2022

ajwerner commented Nov 15, 2022

jordanlewis commented Nov 16, 2022 •

edited

Loading

postamar commented Nov 22, 2022

catalog: v22.1.9: referencing table was dropped without dependency unlinking #91645

catalog: v22.1.9: referencing table was dropped without dependency unlinking #91645

Comments

cockroach-teamcity commented Nov 10, 2022 • edited by cockroach-jira-scripts Loading

ajwerner commented Nov 15, 2022

ajwerner commented Nov 15, 2022

jordanlewis commented Nov 16, 2022 • edited Loading

postamar commented Nov 22, 2022

cockroach-teamcity commented Nov 10, 2022 •

edited by cockroach-jira-scripts

Loading

jordanlewis commented Nov 16, 2022 •

edited

Loading