sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #85212

cockroach-teamcity · 2022-07-28T11:12:06Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3461050768/?referrer=webhooks_plugin

Panic message:

error.go:88: unexpected error from the vectorized engine: context canceled
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:338
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:200
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:378
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*OrderedSynchronizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/ordered_synchronizer.eg.go:135
| github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:377
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*limitOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/limit.go:47
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:239
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:243
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:257
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:548
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1289
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1381
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1072
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:709
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1705
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1707
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1629
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:668
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:648
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2)
Wraps: (3) tags: [n1,client=172.17.54.24:54910,hostssl,user=×,streamID=×,received-error]
Wraps: (4) tags: [n×,streamID=×,sent-error=×]
Wraps: (5)
Wraps: (6) assertion failure
Wraps: (7)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:353
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash.(*HashTable).FullBuild
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash/hashtable.go:361
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).build
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:318
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:280
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:198
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:196
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:125
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*unorderedDistinct).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/unordered_distinct.go:95
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:198
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:196
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).processNextBatch
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:639
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1.1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:563
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:601
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:558
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:793
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1320
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (8) unexpected error from the vectorized engine
Wraps: (9) context canceled
| -- cause hidden behind barrier
| context canceled
| (1) context canceled
| Error types: (1) *errors.errorString
Error types: (1) *withstack.withStack (2) *colexecerror.notInternalError (3) *contexttags.withContext (4) *contexttags....

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 337 to 339 in 11787ed

    
           i.close() 
        
           err := logcrash.PanicAsError(0, panicObj) 
        
           log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

in pkg/sql/colflow/colrpc.(*Inbox).Next.func1
/usr/local/go/src/runtime/panic.go#L964-L966 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 199 to 201 in 11787ed

    
           func ExpectedError(err error) { 
        
           	panic(newNotInternalError(err)) 
        
           }

in pkg/sql/colexecerror.ExpectedError

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 377 to 379 in 11787ed

    
           	// and returned in DrainMeta. 
        
           	colexecerror.ExpectedError(meta.Err) 
        
           }

in pkg/sql/colflow/colrpc.(*Inbox).Next

cockroach/pkg/sql/colexec/ordered_synchronizer.eg.go

Lines 134 to 136 in 11787ed

    
           for i := range o.inputs { 
        
           	o.inputBatches[i] = o.inputs[i].Root.Next() 
        
           	o.updateComparators(i)

in pkg/sql/colexec.(*OrderedSynchronizer).Next

cockroach/pkg/sql/colexecop/operator.go

Lines 376 to 378 in 11787ed

    
           func (n *noopOperator) Next() coldata.Batch { 
        
           	return n.Input.Next() 
        
           }

in pkg/sql/colexecop.(*noopOperator).Next

cockroach/pkg/sql/colexec/limit.go

Lines 46 to 48 in 11787ed

    
           } 
        
           bat := c.Input.Next() 
        
           length := bat.Length()

in pkg/sql/colexec.(*limitOp).Next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 238 to 240 in 11787ed

    
           func (f *BatchFlowCoordinator) nextAdapter() { 
        
           	f.batch = f.input.Root.Next() 
        
           }

in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in 11787ed

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 242 to 244 in 11787ed

    
           func (f *BatchFlowCoordinator) next() error { 
        
           	return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter) 
        
           }

in pkg/sql/colflow.(*BatchFlowCoordinator).next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 281 to 283 in 11787ed

    
           for status == execinfra.NeedMoreRows { 
        
           	err := f.next() 
        
           	if err != nil {

in pkg/sql/colflow.(*BatchFlowCoordinator).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 256 to 258 in 11787ed

    
           	log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine") 
        
           	f.batchFlowCoordinator.Run(ctx) 
        
           }

in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 547 to 549 in 11787ed

    
           // TODO(radu): this should go through the flow scheduler. 
        
           flow.Run(ctx, func() {})

in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1288 to 1290 in 11787ed

    
           recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows) 
        
           runCleanup := dsp.Run(planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */) 
        
           return func() {

in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/conn_executor_exec.go

Lines 1380 to 1382 in 11787ed

    
           // the planner whether or not to plan remote table readers. 
        
           cleanup := ex.server.cfg.DistSQLPlanner.PlanAndRun( 
        
           	ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,

in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1071 to 1073 in 11787ed

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err := ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distributePlan.WillDistribute(), progAtomic,

in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 708 to 710 in 11787ed

    
           if err := ex.dispatchToExecutionEngine(ctx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 128 to 130 in 11787ed

    
           } else { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res) 
        
           }

in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor.go

Lines 1704 to 1706 in 11787ed

    
           ev, payload, err = ex.execStmt(ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes) 
        
           return err

in pkg/sql.(*connExecutor).execCmd.func1

cockroach/pkg/sql/conn_executor.go

Lines 1706 to 1708 in 11787ed

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1628 to 1630 in 11787ed

    
           var err error 
        
           if err = ex.execCmd(ex.Ctx()); err != nil { 
        
           	if errors.IsAny(err, io.EOF, errDrainingComplete) {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 667 to 669 in 11787ed

    
           	}() 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 647 to 649 in 11787ed

    
           	reservedOwned = false // We're about to pass ownership away. 
        
           	retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn) 
        
           }()

in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
/usr/local/go/src/runtime/asm_amd64.s#L1370-L1372 in runtime.goexit

pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 338
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 965
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 200
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 378
pkg/sql/colexec/ordered_synchronizer.eg.go in pkg/sql/colexec.(*OrderedSynchronizer).Next at line 135
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 377
pkg/sql/colexec/limit.go in pkg/sql/colexec.(*limitOp).Next at line 47
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 239
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 243
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 282
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 257
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 548
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1289
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1381
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1072
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 709
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 129
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 1705
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1707
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1629
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 668
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 648
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1371

Tag	Value
Cockroach Release	`v21.2.9`
Cockroach SHA:	`11787ed`
Platform	linux amd64
Distribution	CCL
Environment	v21.2.9
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-18106

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-08-15T16:47:23Z

dup of #82947

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jul 28, 2022

yuzefovich closed this as completed Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #85212

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #85212

cockroach-teamcity commented Jul 28, 2022 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Aug 15, 2022

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #85212

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1 ... #85212

Comments

cockroach-teamcity commented Jul 28, 2022 • edited by cockroach-jira-scripts Loading

yuzefovich commented Aug 15, 2022

cockroach-teamcity commented Jul 28, 2022 •

edited by cockroach-jira-scripts

Loading