sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colexece... #84061

cockroach-teamcity · 2022-07-08T08:32:46Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3410370442/?referrer=webhooks_plugin

Panic message:

error.go:88: unexpected error from the vectorized engine: context canceled
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:363
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*unorderedDistinct).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/unordered_distinct.go:95
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:198
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:196
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:125
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:261
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:286
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:292
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:127
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).nextAdapter
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:144
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:149
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:186
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:732
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:447
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:244
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:548
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).planAndRunSubquery
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1180
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1085
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1372
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1072
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:709
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1705
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1707
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1629
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:668
Wraps: (3) unexpected error from the vectorized engine
Wraps: (4) context canceled
| -- cause hidden behind barrier
| context canceled
| (1) context canceled
| Error types: (1) *errors.errorString
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.withPrefix (4) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: context canceled
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:88: *withstack.withStack (top exception)
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colexecerror/error.go

Lines 87 to 89 in 11787ed

    
           	// unexpected. 
        
           	retErr = errors.NewAssertionErrorWithWrappedErrf(err, "unexpected error from the vectorized engine") 
        
           }

in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
/usr/local/go/src/runtime/panic.go#L964-L966 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 191 to 193 in 11787ed

    
           func InternalError(err error) { 
        
           	panic(err) 
        
           }

in pkg/sql/colexecerror.InternalError

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 362 to 364 in 11787ed

    
           if err != nil { 
        
           	colexecerror.InternalError(err) 
        
           }

in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next

cockroach/pkg/sql/colexec/unordered_distinct.go

Lines 94 to 96 in 11787ed

    
           for { 
        
           	op.lastInputBatch = op.Input.Next() 
        
           	op.lastInputBatchOrigLen = op.lastInputBatch.Length()

in pkg/sql/colexec.(*unorderedDistinct).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 197 to 199 in 11787ed

    
           func() { 
        
           	batch = d.inMemoryOp.Next() 
        
           },

in pkg/sql/colexec.(*diskSpillerBase).Next.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in 11787ed

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 195 to 197 in 11787ed

    
           var batch coldata.Batch 
        
           if err := colexecerror.CatchVectorizedRuntimeError( 
        
           	func() {

in pkg/sql/colexec.(*diskSpillerBase).Next

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 124 to 126 in 11787ed

    
           func (d *simpleProjectOp) Next() coldata.Batch { 
        
           	batch := d.Input.Next() 
        
           	if batch.Length() == 0 {

in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next

cockroach/pkg/sql/colexec/materializer.go

Lines 260 to 262 in 11787ed

    
           // Get a fresh batch. 
        
           m.batch = m.input.Next() 
        
           if m.batch.Length() == 0 {

in pkg/sql/colexec.(*Materializer).next

cockroach/pkg/sql/colexec/materializer.go

Lines 285 to 287 in 11787ed

    
           func (m *Materializer) nextAdapter() { 
        
           	m.outputRow = m.next() 
        
           }

in pkg/sql/colexec.(*Materializer).nextAdapter

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in 11787ed

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colexec/materializer.go

Lines 291 to 293 in 11787ed

    
           for m.State == execinfra.StateRunning { 
        
           	if err := colexecerror.CatchVectorizedRuntimeError(m.nextAdapter); err != nil { 
        
           		m.MoveToDraining(err)

in pkg/sql/colexec.(*Materializer).Next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 126 to 128 in 11787ed

    
           if f.State == execinfra.StateRunning { 
        
           	row, meta := f.input.Next() 
        
           	if meta != nil {

in pkg/sql/colflow.(*FlowCoordinator).next

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 143 to 145 in 11787ed

    
           func (f *FlowCoordinator) nextAdapter() { 
        
           	f.row, f.meta = f.next() 
        
           }

in pkg/sql/colflow.(*FlowCoordinator).nextAdapter

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in 11787ed

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 148 to 150 in 11787ed

    
           func (f *FlowCoordinator) Next() (rowenc.EncDatumRow, *execinfrapb.ProducerMetadata) { 
        
           	if err := colexecerror.CatchVectorizedRuntimeError(f.nextAdapter); err != nil { 
        
           		f.MoveToDraining(err)

in pkg/sql/colflow.(*FlowCoordinator).Next

cockroach/pkg/sql/execinfra/base.go

Lines 185 to 187 in 11787ed

    
           for { 
        
           	row, meta := src.Next() 
        
           	// Emit the row; stop if no more rows are needed.

in pkg/sql/execinfra.Run

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 731 to 733 in 11787ed

    
           	pb.self.Start(ctx) 
        
           	Run(pb.Ctx, pb.self, pb.Output) 
        
           }

in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run

cockroach/pkg/sql/flowinfra/flow.go

Lines 446 to 448 in 11787ed

    
           	log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc) 
        
           	headProc.Run(ctx) 
        
           }

in pkg/sql/flowinfra.(*FlowBase).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 243 to 245 in 11787ed

    
           // as the root, so we run this flow with the default implementation. 
        
           f.FlowBase.Run(ctx, doneFn) 
        
           return

in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 547 to 549 in 11787ed

    
           // TODO(radu): this should go through the flow scheduler. 
        
           flow.Run(ctx, func() {})

in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1179 to 1181 in 11787ed

    
           subqueryPlans[planIdx].started = true 
        
           dsp.Run(subqueryPlanCtx, planner.txn, subqueryPhysPlan, subqueryRecv, evalCtx, nil /* finishedSetupFn */)() 
        
           if err := subqueryRowReceiver.Err(); err != nil {

in pkg/sql.(*DistSQLPlanner).planAndRunSubquery

cockroach/pkg/sql/distsql_running.go

Lines 1084 to 1086 in 11787ed

    
           for planIdx, subqueryPlan := range subqueryPlans { 
        
           	if err := dsp.planAndRunSubquery( 
        
           		ctx,

in pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries

cockroach/pkg/sql/conn_executor_exec.go

Lines 1371 to 1373 in 11787ed

    
           defer subqueryResultMemAcc.Close(ctx) 
        
           if !ex.server.cfg.DistSQLPlanner.PlanAndRunSubqueries( 
        
           	ctx, planner, evalCtxFactory, planner.curPlan.subqueryPlans, recv, &subqueryResultMemAcc,

in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1071 to 1073 in 11787ed

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err := ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distributePlan.WillDistribute(), progAtomic,

in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 708 to 710 in 11787ed

    
           if err := ex.dispatchToExecutionEngine(ctx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 128 to 130 in 11787ed

    
           } else { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res) 
        
           }

in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor.go

Lines 1704 to 1706 in 11787ed

    
           ev, payload, err = ex.execStmt(ctx, tcmd.Statement, nil /* prepared */, nil /* pinfo */, stmtRes) 
        
           return err

in pkg/sql.(*connExecutor).execCmd.func1

cockroach/pkg/sql/conn_executor.go

Lines 1706 to 1708 in 11787ed

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1628 to 1630 in 11787ed

    
           var err error 
        
           if err = ex.execCmd(ex.Ctx()); err != nil { 
        
           	if errors.IsAny(err, io.EOF, errDrainingComplete) {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 667 to 669 in 11787ed

    
           	}() 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

in pkg/sql.(*Server).ServeConn

pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 88
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 965
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 192
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next at line 363
pkg/sql/colexec/unordered_distinct.go in pkg/sql/colexec.(*unorderedDistinct).Next at line 95
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next.func1 at line 198
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next at line 196
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 125
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 261
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 286
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 292
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).next at line 127
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).nextAdapter at line 144
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Next at line 149
pkg/sql/execinfra/base.go in pkg/sql/execinfra.Run at line 186
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 732
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 447
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 244
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 548
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).planAndRunSubquery at line 1180
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunSubqueries at line 1085
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1372
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1072
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 709
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 129
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 1705
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1707
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1629
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 668

Tag	Value
Cockroach Release	`v21.2.9`
Cockroach SHA:	`11787ed`
Platform	linux amd64
Distribution	CCL
Environment	v21.2.9
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-17437

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-08-15T16:48:33Z

dup of #82947

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jul 8, 2022

yuzefovich closed this as completed Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colexece... #84061

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colexece... #84061

cockroach-teamcity commented Jul 8, 2022 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Aug 15, 2022

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colexece... #84061

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colexece... #84061

Comments

cockroach-teamcity commented Jul 8, 2022 • edited by cockroach-jira-scripts Loading

yuzefovich commented Aug 15, 2022

cockroach-teamcity commented Jul 8, 2022 •

edited by cockroach-jira-scripts

Loading