Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: crash_reporting.go:376: internal-planner.×.×: unexpected 10240 leftover bytes (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPa... #105321

Closed
cockroach-sentry opened this issue Jun 21, 2023 · 1 comment
Closed

Sentry: crash_reporting.go:376: internal-planner.×.×: unexpected 10240 leftover bytes (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPa... #105321

cockroach-sentry opened this issue Jun 21, 2023 · 1 comment

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Jun 21, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4266666283/?referrer=webhooks_plugin

Panic Message:

crash_reporting.go:376: internal-planner.×.×: unexpected 10240 leftover bytes
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
  | 	github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:376
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop
  | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:445
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop
  | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:420
  | github.com/cockroachdb/cockroach/pkg/sql.newInternalPlanner.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/planner.go:426
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:835
  | github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).ComputeAggregatedTs
  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/flush.go:332
  | github.com/cockroachdb/cockroach/pkg/sql.glob..func85
  | 	github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:5644
  | github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).getPlanInfo.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:590
  | github.com/cockroachdb/cockroach/pkg/sql.constructVirtualScan.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_factory_util.go:250
  | github.com/cockroachdb/cockroach/pkg/sql.(*delayedNode).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/delayed.go:50
  | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  | github.com/cockroachdb/cockroach/pkg/sql.startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:521
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:163
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:187
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*SerialUnorderedSynchronizer).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/serial_unordered_synchronizer.go:72
  | github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*OneInputInitCloserHelper).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:375
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:268
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:267
  | github.com/cockroachdb/cockroach/pkg/sql.(*rowSourceToPlanNode).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/row_source_to_plan_node.go:72
Wraps: (2) internal-planner.×.×: unexpected 10240 leftover bytes
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: internal-planner.×.×: unexpected 10240 leftover bytes
crash_reporting.go:376: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/row_source_to_plan_node.go

Lines 71 to 73 in d5a076f

func (r *rowSourceToPlanNode) startExec(params runParams) error {
r.source.Start(params.ctx)
return nil

cockroach/pkg/sql/colexec/materializer.go

Lines 266 to 268 in d5a076f

// reached), so we need to wrap it with a catcher.
if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in d5a076f

}()
operation()
return retErr

cockroach/pkg/sql/colexec/materializer.go

Lines 267 to 269 in d5a076f

if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
}); err != nil {

cockroach/pkg/sql/colexecop/operator.go

Lines 374 to 376 in d5a076f

}
h.Input.Init(h.Ctx)
}

cockroach/pkg/sql/colexec/serial_unordered_synchronizer.go

Lines 71 to 73 in d5a076f

for _, input := range s.inputs {
input.Root.Init(s.Ctx)
}

cockroach/pkg/sql/colexec/columnarizer.go

Lines 186 to 188 in d5a076f

ctx = c.StartInternal(ctx, "columnarizer" /* name */)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 162 to 164 in d5a076f

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)

cockroach/pkg/sql/plan.go

Lines 520 to 522 in d5a076f

}
return walkPlan(params.ctx, plan, o)
}

cockroach/pkg/sql/walk.go

Lines 42 to 44 in d5a076f

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err

cockroach/pkg/sql/walk.go

Lines 78 to 80 in d5a076f

}
v.visitInternal(plan, name)
return plan

cockroach/pkg/sql/walk.go

Lines 296 to 298 in d5a076f

}
}

cockroach/pkg/sql/walk.go

Lines 111 to 113 in d5a076f

}
v.err = v.observer.leaveNode(name, plan)
}()

cockroach/pkg/sql/plan.go

Lines 517 to 519 in d5a076f

}
return n.startExec(params)
},

cockroach/pkg/sql/delayed.go

Lines 49 to 51 in d5a076f

plan, err := d.constructor(params.ctx, params.p)
if err != nil {

cockroach/pkg/sql/exec_factory_util.go

Lines 249 to 251 in d5a076f

constructor: func(ctx context.Context, p *planner) (planNode, error) {
return constructor(ctx, p, tn.Catalog())
},

cockroach/pkg/sql/virtual_schema.go

Lines 589 to 591 in d5a076f

if def.generator != nil && !def.preferIndexOverGenerator(p, index, idxConstraint) {
next, cleanup, err := def.generator(ctx, p, dbDesc, stopper)
if err != nil {

cockroach/pkg/sql/crdb_internal.go

Lines 5643 to 5645 in d5a076f

s := p.extendedEvalCtx.statsProvider
curAggTs := s.ComputeAggregatedTs()
aggInterval := s.GetAggregationInterval()

https://github.com/cockroachdb/cockroach/blob/d5a076f2953c0712193b8cbe08bbb682563c5675/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/flush.go#L331-L333
GOROOT/src/runtime/signal_unix.go#L834-L836
GOROOT/src/runtime/panic.go#L259-L261
GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in d5a076f

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in d5a076f

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colexecerror/error.go

Lines 57 to 59 in d5a076f

if !shouldCatchPanic(panicEmittedFrom) {
panic(panicObj)
}

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/planner.go

Lines 425 to 427 in d5a076f

// Stop the memory monitor.
plannerMon.Stop(ctx)
}

cockroach/pkg/util/mon/bytes_usage.go

Lines 419 to 421 in d5a076f

func (mm *BytesMonitor) Stop(ctx context.Context) {
mm.doStop(ctx, true)
}

cockroach/pkg/util/mon/bytes_usage.go

Lines 444 to 446 in d5a076f

if check && mm.mu.curAllocated != 0 {
logcrash.ReportOrPanic(
ctx, &mm.settings.SV,

cockroach/pkg/util/log/logcrash/crash_reporting.go

Lines 375 to 377 in d5a076f

) {
err := errors.Newf(format, reportables...)
if !build.IsRelease() || (sv != nil && PanicOnAssertions.Get(sv)) { 

pkg/sql/row_source_to_plan_node.go in pkg/sql.(*rowSourceToPlanNode).startExec at line 72
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start at line 267
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start.func1 at line 268
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*OneInputInitCloserHelper).Init at line 375
pkg/sql/colexec/serial_unordered_synchronizer.go in pkg/sql/colexec.(*SerialUnorderedSynchronizer).Init at line 72
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 187
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 163
pkg/sql/plan.go in pkg/sql.startExec at line 521
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 518
pkg/sql/delayed.go in pkg/sql.(*delayedNode).startExec at line 50
pkg/sql/exec_factory_util.go in pkg/sql.constructVirtualScan.func1 at line 250
pkg/sql/virtual_schema.go in pkg/sql.(*virtualDefEntry).getPlanInfo.func1 at line 590
pkg/sql/crdb_internal.go in pkg/sql.glob..func85 at line 5644
pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/flush.go in pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).ComputeAggregatedTs at line 332
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 835
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 58
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/planner.go in pkg/sql.newInternalPlanner.func1 at line 426
pkg/util/mon/bytes_usage.go in pkg/util/mon.(*BytesMonitor).Stop at line 420
pkg/util/mon/bytes_usage.go in pkg/util/mon.(*BytesMonitor).doStop at line 445
pkg/util/log/logcrash/crash_reporting.go in pkg/util/log/logcrash.ReportOrPanic at line 376

Tags

Tag Value
Command demo
Environment v22.2.6
Go Version go1.19.4
Platform darwin amd64
Distribution CCL
Cockroach Release v22.2.6
Cockroach SHA d5a076f
# of CPUs 16
# of Goroutines 430

Jira issue: CRDB-28978
@yuzefovich
Copy link
Member

fixed by #105324

@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: crash_reporting.go:376: internal-planner.×.×: unexpected 10240 leftover bytes (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPa... Sentry: crash_reporting.go:376: internal-planner.×.×: unexpected 10240 leftover bytes (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPa... Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry