Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #83029

Closed
cockroach-teamcity opened this issue Jun 17, 2022 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jun 17, 2022

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3356489762/?referrer=webhooks_plugin

Panic message:

error.go:88: unexpected error from the vectorized engine: context canceled
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:338
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:200
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:378
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*OrderedSynchronizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/ordered_synchronizer.eg.go:135
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:377
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*limitOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/limit.go:47
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:239
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:243
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:258
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:548
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1289
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1381
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1072
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:709
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:216
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1781
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1783
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1637
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:668
Wraps: (2)
Wraps: (3) tags: [n24,client=10.42.244.0:37209,hostssl,user=×,streamID=×,received-error]
Wraps: (4) tags: [n×,streamID=×,sent-error=×]
Wraps: (5)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:338
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:200
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:378
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.func1.1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:185
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:249
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (6)
Wraps: (7) tags: [n×,streamID=×,received-error=×]
Wraps: (8) tags: [n×,streamID=×,sent-error=×]
Wraps: (9)
Wraps: (10) assertion failure
Wraps: (11)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*externalSorter).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/external_sort.go:362
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:211
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*fnOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/fn_op.go:32
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase....

Stacktrace (expand for inline code snippets):

i.close()
err := logcrash.PanicAsError(0, panicObj)
log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)
in pkg/sql/colflow/colrpc.(*Inbox).Next.func1
/usr/local/go/src/runtime/panic.go#L964-L966 in runtime.gopanic
func ExpectedError(err error) {
panic(newNotInternalError(err))
}
in pkg/sql/colexecerror.ExpectedError
// and returned in DrainMeta.
colexecerror.ExpectedError(meta.Err)
}
in pkg/sql/colflow/colrpc.(*Inbox).Next
func (bic *batchInfoCollector) next() {
bic.batch = bic.Operator.Next()
}
in pkg/sql/colflow.(*batchInfoCollector).next
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()
in pkg/sql/colflow.(*batchInfoCollector).Next
for i := range o.inputs {
o.inputBatches[i] = o.inputs[i].Root.Next()
o.updateComparators(i)
in pkg/sql/colexec.(*OrderedSynchronizer).Next
func (bic *batchInfoCollector) next() {
bic.batch = bic.Operator.Next()
}
in pkg/sql/colflow.(*batchInfoCollector).next
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()
in pkg/sql/colflow.(*batchInfoCollector).Next
func (n *noopOperator) Next() coldata.Batch {
return n.Input.Next()
}
in pkg/sql/colexecop.(*noopOperator).Next
}
bat := c.Input.Next()
length := bat.Length()
in pkg/sql/colexec.(*limitOp).Next
func (bic *batchInfoCollector) next() {
bic.batch = bic.Operator.Next()
}
in pkg/sql/colflow.(*batchInfoCollector).next
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()
in pkg/sql/colflow.(*batchInfoCollector).Next
func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}
in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError
func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}
in pkg/sql/colflow.(*BatchFlowCoordinator).next
for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {
in pkg/sql/colflow.(*BatchFlowCoordinator).Run
log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}
in pkg/sql/colflow.(*vectorizedFlow).Run
// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run
recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun
// the planner whether or not to plan remote table readers.
cleanup := ex.server.cfg.DistSQLPlanner.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*connExecutor).execWithDistSQLEngine
ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err := ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distributePlan.WillDistribute(), progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine
if err := ex.dispatchToExecutionEngine(ctx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState
} else {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res)
}
in pkg/sql.(*connExecutor).execStmt
}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal
res = stmtRes
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo)
return err
in pkg/sql.(*connExecutor).execCmd.func2
return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd
var err error
if err = ex.execCmd(ex.Ctx()); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run
}()
return h.ex.run(ctx, s.pool, reserved, cancel)
}
in pkg/sql.(*Server).ServeConn

pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 338
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 965
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 200
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 378
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colexec/ordered_synchronizer.eg.go in pkg/sql/colexec.(*OrderedSynchronizer).Next at line 135
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 377
pkg/sql/colexec/limit.go in pkg/sql/colexec.(*limitOp).Next at line 47
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 239
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 243
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 282
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 258
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 548
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1289
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1381
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1072
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 709
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 129
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 216
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1781
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1783
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1637
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 668
Tag Value
Cockroach Release v21.2.11
Cockroach SHA: b05bab3
Platform linux amd64
Distribution CCL
Environment v21.2.11
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-16804

@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jun 17, 2022
@yuzefovich
Copy link
Member

dup of #82947

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

2 participants