sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) assertion failure Wraps: (4) attached stack trace -- stack trace: | github.com/cockroachdb/cock...

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3363791563/?referrer=webhooks_plugin

Panic message:

error.go:88: unexpected error from the vectorized engine: context canceled
(1)
Wraps: (2)
Wraps: (3) assertion failure
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*hashBasedPartitioner).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/hash_based_partitioner.go:442
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:211
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:125
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).processNextBatch
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:639
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1.1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:563
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:601
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:558
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:799
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1326
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (5) unexpected error from the vectorized engine
Wraps: (6) context canceled
| -- cause hidden behind barrier
| context canceled
| (1) context canceled
| Error types: (1) *errors.errorString
Error types: (1) *colexecerror.notInternalError (2) *colexecerror.notInternalError (3) *assert.withAssertionFailure (4) *withstack.withStack (5) *errutil.withPrefix (6) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: context canceled
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:88: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colexecerror/error.go

Lines 87 to 89 in b05bab3

	// unexpected.
	retErr = errors.NewAssertionErrorWithWrappedErrf(err, "unexpected error from the vectorized engine")
	}

in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
/usr/local/go/src/runtime/panic.go#L964-L966 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 191 to 193 in b05bab3

	func InternalError(err error) {
	panic(err)
	}

in pkg/sql/colexecerror.InternalError

cockroach/pkg/sql/colexec/hash_based_partitioner.go

Lines 441 to 443 in b05bab3

	if err := op.fdState.fdSemaphore.Acquire(op.Ctx, toAcquire); err != nil {
	colexecerror.InternalError(err)
	}

in pkg/sql/colexec.(*hashBasedPartitioner).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 210 to 212 in b05bab3

	d.diskBackedOpInitialized = true
	return d.diskBackedOp.Next()
	}

in pkg/sql/colexec.(*diskSpillerBase).Next

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 124 to 126 in b05bab3

	func (d *simpleProjectOp) Next() coldata.Batch {
	batch := d.Input.Next()
	if batch.Length() == 0 {

in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next

cockroach/pkg/sql/colflow/stats.go

Lines 97 to 99 in b05bab3

	func (bic *batchInfoCollector) next() {
	bic.batch = bic.Operator.Next()
	}

in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

	}()
	operation()
	return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 105 to 107 in b05bab3

	// execution time (e.g. in the statement bundle).
	err := colexecerror.CatchVectorizedRuntimeError(bic.next)
	bic.stopwatch.Stop()

in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colflow/routers.go

Lines 638 to 640 in b05bab3

	func (r *HashRouter) processNextBatch(ctx context.Context) bool {
	b := r.Input.Next()
	n := b.Length()

in pkg/sql/colflow.(*HashRouter).processNextBatch

cockroach/pkg/sql/colflow/routers.go

Lines 562 to 564 in b05bab3

	processNextBatch := func() {
	done = r.processNextBatch(ctx)
	}

in pkg/sql/colflow.(*HashRouter).Run.func1.1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

	}()
	operation()
	return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/routers.go

Lines 600 to 602 in b05bab3

	if err := colexecerror.CatchVectorizedRuntimeError(processNextBatch); err != nil {
	r.cancelOutputs(ctx, err)

in pkg/sql/colflow.(*HashRouter).Run.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

	}()
	operation()
	return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/routers.go

Lines 557 to 559 in b05bab3

	// well for more fine-grained control of error propagation.
	if err := colexecerror.CatchVectorizedRuntimeError(func() {
	r.Input.Init(ctx)

in pkg/sql/colflow.(*HashRouter).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 798 to 800 in b05bab3

	runRouter := func(ctx context.Context, _ context.CancelFunc) {
	router.Run(logtags.AddTag(ctx, "hashRouterID", streamIDs))
	}

in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 1325 to 1327 in b05bab3

	go func() {
	run(ctx, flowCtxCancel)
	if wg != nil {

in pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1
/usr/local/go/src/runtime/asm_amd64.s#L1370-L1372 in runtime.goexit

pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 88
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 965
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 192
pkg/sql/colexec/hash_based_partitioner.go in pkg/sql/colexec.(*hashBasedPartitioner).Next at line 442
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next at line 211
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 125
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).processNextBatch at line 639
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1.1 at line 563
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1 at line 601
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run at line 558
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1 at line 799
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1 at line 1326
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1371

Tag	Value
Cockroach Release	v21.2.11
Cockroach SHA:	b05bab3
Platform	linux amd64
Distribution	CCL
Environment	v21.2.11
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-16867

[yuzefovich]

dup of #82947