Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: errors.go:83: referenced descriptor ID 88: descriptor not found (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/catalog/int... #106556

Closed
cockroach-sentry opened this issue Jul 10, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Jul 10, 2023

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4306432386/?referrer=webhooks_plugin

Panic Message:

errors.go:83: referenced descriptor ID 88: descriptor not found
(1) assertion failure
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.wrapError
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:128
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.processDescriptorResultRow
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:111
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.query
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:73
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogReader.GetDescriptorEntries
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_reader.go:224
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.(*StoredCatalog).EnsureFromStorageByIDs
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/stored_catalog.go:308
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.getDescriptorsByID
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:191
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getDescriptorByID
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:88
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getDescriptorByName
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:336
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getSchemaByName
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/schema.go:73
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetImmutableSchemaByName
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/schema.go:61
  | github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).LookupSchema
  | 	github.com/cockroachdb/cockroach/pkg/sql/schema_resolver.go:132
  | github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).ResolveFunction
  | 	github.com/cockroachdb/cockroach/pkg/sql/schema_resolver.go:416
  | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveFunction
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:335
  | github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:450
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/memo.go:372
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:530
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:232
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1473
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1084
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:694
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2438
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2123
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2127
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2048
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:881
  | [...repeated from below...]
Wraps: (3) referenced descriptor ID 88
Wraps: (4) secondary error attachment
  | failed to find descriptor [88]
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.requiredError
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:172
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.build
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:142
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.processDescriptorResultRow
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:109
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.query
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_query.go:73
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogReader.GetDescriptorEntries
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/catalog_reader.go:224
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.(*StoredCatalog).EnsureFromStorageByIDs
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv/stored_catalog.go:308
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.getDescriptorsByID
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:191
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getDescriptorByID
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:88
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getDescriptorByName
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/descriptor.go:336
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).getSchemaByName
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/schema.go:73
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetImmutableSchemaByName
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/schema.go:61
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).LookupSchema
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/schema_resolver.go:132
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*schemaResolver).ResolveFunction
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/schema_resolver.go:416
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveFunction
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:335
  |   | github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:450
  |   | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/memo.go:372
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:530
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:232
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1473
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1084
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:694
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  |   | 	github.com/cockroach...
Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1593-L1595

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}

var err error
if err = ex.execCmd(); err != nil {
// Both of these errors are normal ways for the connExecutor to exit.

return err
}()
// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

canAutoCommit := ex.implicitTxn() && (tcmd.LastInBatch || !implicitTxnForBatch)
ev, payload, err = ex.execStmt(
ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes, canAutoCommit,

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)

} else {
err = op(ctx)
}

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

// between here and there needs to happen even if there's an error.
err := ex.makeExecPlan(ctx, planner)
// We'll be closing the plan manually below after execution; this

func (ex *connExecutor) makeExecPlan(ctx context.Context, planner *planner) error {
if err := planner.makeOptimizerPlan(ctx); err != nil {
log.VEventf(ctx, 1, "optimizer plan failed: %v", err)

execMemo, err := opc.buildExecMemo(ctx)
if err != nil {

if ok {
if isStale, err := cachedData.Memo.IsStale(ctx, p.EvalContext(), &opc.catalog); err != nil {
return nil, err

// access the object.
if depsUpToDate, err := m.Metadata().CheckDependencies(ctx, evalCtx, catalog); err != nil {
return true, err

for name := range md.builtinRefsByName {
definition, err := optCatalog.ResolveFunction(
ctx, &name, &evalCtx.SessionData().SearchPath,

) (*tree.ResolvedFunctionDefinition, error) {
return oc.planner.ResolveFunction(ctx, name, path)
}

schema := path.GetSchema(i)
found, prefix, err := sr.LookupSchema(ctx, sr.CurrentDatabase(), schema)
if err != nil {

}
sc, err := sr.descCollection.GetImmutableSchemaByName(ctx, sr.txn, db, scName, flags)
if err != nil || sc == nil {

flags.RequireMutable = false
return tc.getSchemaByName(ctx, txn, db, scName, flags)
}

) (catalog.SchemaDescriptor, error) {
desc, err := tc.getDescriptorByName(ctx, txn, db, nil /* sc */, schemaName, flags, catalog.Schema)
if err != nil {

desc, err := tc.getDescriptorByID(ctx, txn, flags, id)
if err != nil {

var arr [1]catalog.Descriptor
if err := getDescriptorsByID(
ctx, tc, txn, flags, arr[:], ids...,

if !readIDs.Empty() {
if err = tc.stored.EnsureFromStorageByIDs(ctx, txn, readIDs, catalog.Any); err != nil {
return err

}
c, err := sc.GetDescriptorEntries(ctx, txn, ids.Ordered(), true /* isRequired */, descriptorType)
if err != nil {

}
err := cq.query(ctx, txn, &mc, func(codec keys.SQLCodec, b *kv.Batch) {
for _, id := range ids {

case keys.DescriptorTableID:
err = cq.processDescriptorResultRow(row, out)
default:

if err != nil {
return wrapError(expectedType, id, err)
}

}
return errors.Wrapf(err, "referenced descriptor ID %d", id)
}

src/runtime/asm_amd64.s#L1593-L1595
GOROOT/src/runtime/proc.go#L232-L234
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6324-L6326
GOROOT/src/runtime/proc.go#L6347-L6349
// found with the given id.
var ErrDescriptorNotFound = errors.New("descriptor not found")

src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 758
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 881
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2048
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2127
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2123
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 130
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2438
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 131
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 694
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1084
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 1473
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 232
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 530
pkg/sql/opt/memo/memo.go in pkg/sql/opt/memo.(*Memo).IsStale at line 372
pkg/sql/opt/metadata.go in pkg/sql/opt.(*Metadata).CheckDependencies at line 450
pkg/sql/opt_catalog.go in pkg/sql.(*optCatalog).ResolveFunction at line 335
pkg/sql/schema_resolver.go in pkg/sql.(*schemaResolver).ResolveFunction at line 416
pkg/sql/schema_resolver.go in pkg/sql.(*schemaResolver).LookupSchema at line 132
pkg/sql/catalog/descs/schema.go in pkg/sql/catalog/descs.(*Collection).GetImmutableSchemaByName at line 61
pkg/sql/catalog/descs/schema.go in pkg/sql/catalog/descs.(*Collection).getSchemaByName at line 73
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getDescriptorByName at line 336
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.(*Collection).getDescriptorByID at line 88
pkg/sql/catalog/descs/descriptor.go in pkg/sql/catalog/descs.getDescriptorsByID at line 191
pkg/sql/catalog/internal/catkv/stored_catalog.go in pkg/sql/catalog/internal/catkv.(*StoredCatalog).EnsureFromStorageByIDs at line 308
pkg/sql/catalog/internal/catkv/catalog_reader.go in pkg/sql/catalog/internal/catkv.catalogReader.GetDescriptorEntries at line 224
pkg/sql/catalog/internal/catkv/catalog_query.go in pkg/sql/catalog/internal/catkv.catalogQuery.query at line 73
pkg/sql/catalog/internal/catkv/catalog_query.go in pkg/sql/catalog/internal/catkv.catalogQuery.processDescriptorResultRow at line 111
pkg/sql/catalog/internal/catkv/catalog_query.go in pkg/sql/catalog/internal/catkv.wrapError at line 128
src/runtime/asm_amd64.s in runtime.goexit at line 1594
GOROOT/src/runtime/proc.go in runtime.main at line 233
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6325
GOROOT/src/runtime/proc.go in runtime.doInit at line 6348
pkg/sql/catalog/errors.go in pkg/sql/catalog.init at line 83

Tags

Tag Value
Command demo
Environment v22.2.9
Go Version go1.19.6
Platform darwin amd64
Distribution CCL
Cockroach Release v22.2.9
Cockroach SHA 1b11ca6
# of CPUs 16
# of Goroutines 430

Jira issue: CRDB-29613

@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Jul 10, 2023
@yuzefovich
Copy link
Member

dup of #105257

@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: errors.go:83: referenced descriptor ID 88: descriptor not found (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/catalog/int... Sentry: errors.go:83: referenced descriptor ID 88: descriptor not found (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/catalog/int... Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

2 participants