sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) Wraps: (4) Wraps: (5) assertion failure Wraps: (6) attached stack trace -- stack trace: | githu... #83481

cockroach-teamcity · 2022-06-28T00:09:31Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3383140862/?referrer=webhooks_plugin

Panic message:

error.go:88: unexpected error from the vectorized engine: context canceled
(1)
Wraps: (2)
Wraps: (3)
Wraps: (4)
Wraps: (5) assertion failure
Wraps: (6) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88
| runtime.gopanic
| /usr/local/go/src/runtime/panic.go:965
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:353
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).exec
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:474
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*hashJoiner).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go:290
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next.func1
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:198
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:196
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*bufferExportingOperator).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:320
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*inputPartitioningOperator).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/external_sort.go:747
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*allSpooler).spool
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:146
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*sortOp).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:283
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*externalSorter).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/external_sort.go:347
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*diskSpillerBase).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:211
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:106
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:261
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:286
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:292
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*inputStatCollector).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/stats.go:67
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinReader).readInput
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinreader.go:664
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinReader).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinreader.go:603
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next
| /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:222
Wraps: (7) unexpected error from the vectorized engine
Wraps: (8) context canceled
| -- cause hidden behind barrier
| context canceled
| (1) context canceled
| Error types: (1) *errors.errorString
Error types: (1) *colexecerror.notInternalError (2) *colexecerror.notInternalError (3) *colexecerror.notInternalError (4) *colexecerror.notInternalError (5) *assert.withAssertionFailure (6) *withstack.withStack (7) *errutil.withPrefix (8) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: context canceled
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:88: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError
*colexecerror.notInternalError
*colexecerror.notInternalError
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colexecerror/error.go

Lines 87 to 89 in b05bab3

    
           	// unexpected. 
        
           	retErr = errors.NewAssertionErrorWithWrappedErrf(err, "unexpected error from the vectorized engine") 
        
           }

in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
/usr/local/go/src/runtime/panic.go#L964-L966 in runtime.gopanic

cockroach/pkg/sql/colexecerror/error.go

Lines 191 to 193 in b05bab3

    
           func InternalError(err error) { 
        
           	panic(err) 
        
           }

in pkg/sql/colexecerror.InternalError

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 352 to 354 in b05bab3

    
           	// which will take care of closing any inputs. 
        
           	colexecerror.InternalError(err) 
        
           }

in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next

cockroach/pkg/sql/colflow/stats.go

Lines 97 to 99 in b05bab3

    
           func (bic *batchInfoCollector) next() { 
        
           	bic.batch = bic.Operator.Next() 
        
           }

in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 105 to 107 in b05bab3

    
           // execution time (e.g. in the statement bundle). 
        
           err := colexecerror.CatchVectorizedRuntimeError(bic.next) 
        
           bic.stopwatch.Stop()

in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go

Lines 473 to 475 in b05bab3

    
           for { 
        
           	batch := hj.inputOne.Next() 
        
           	batchSize := batch.Length()

in pkg/sql/colexec/colexecjoin.(*hashJoiner).exec

cockroach/pkg/sql/colexec/colexecjoin/hashjoiner.go

Lines 289 to 291 in b05bab3

    
           case hjProbing: 
        
           	output := hj.exec() 
        
           	if output.Length() == 0 {

in pkg/sql/colexec/colexecjoin.(*hashJoiner).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 197 to 199 in b05bab3

    
           func() { 
        
           	batch = d.inMemoryOp.Next() 
        
           },

in pkg/sql/colexec.(*diskSpillerBase).Next.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 195 to 197 in b05bab3

    
           var batch coldata.Batch 
        
           if err := colexecerror.CatchVectorizedRuntimeError( 
        
           	func() {

in pkg/sql/colexec.(*diskSpillerBase).Next

cockroach/pkg/sql/colflow/stats.go

Lines 97 to 99 in b05bab3

    
           func (bic *batchInfoCollector) next() { 
        
           	bic.batch = bic.Operator.Next() 
        
           }

in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 105 to 107 in b05bab3

    
           // execution time (e.g. in the statement bundle). 
        
           err := colexecerror.CatchVectorizedRuntimeError(bic.next) 
        
           bic.stopwatch.Stop()

in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 319 to 321 in b05bab3

    
           if b.firstSourceDone { 
        
           	return b.secondSource.Next() 
        
           }

in pkg/sql/colexec.(*bufferExportingOperator).Next

cockroach/pkg/sql/colexec/external_sort.go

Lines 746 to 748 in b05bab3

    
           } 
        
           b := o.Input.Next() 
        
           n := b.Length()

in pkg/sql/colexec.(*inputPartitioningOperator).Next

cockroach/pkg/sql/colexec/sort.go

Lines 145 to 147 in b05bab3

    
           p.spooled = true 
        
           for batch := p.Input.Next(); batch.Length() != 0; batch = p.Input.Next() { 
        
           	p.bufferedTuples.AppendTuples(batch, 0 /* startIdx */, batch.Length())

in pkg/sql/colexec.(*allSpooler).spool

cockroach/pkg/sql/colexec/sort.go

Lines 282 to 284 in b05bab3

    
           case sortSpooling: 
        
           	p.input.spool() 
        
           	p.state = sortSorting

in pkg/sql/colexec.(*sortOp).Next

cockroach/pkg/sql/colexec/external_sort.go

Lines 346 to 348 in b05bab3

    
           case externalSorterNewPartition: 
        
           	b := s.Input.Next() 
        
           	if b.Length() == 0 {

in pkg/sql/colexec.(*externalSorter).Next

cockroach/pkg/sql/colexec/disk_spiller.go

Lines 210 to 212 in b05bab3

    
           	d.diskBackedOpInitialized = true 
        
           	return d.diskBackedOp.Next() 
        
           }

in pkg/sql/colexec.(*diskSpillerBase).Next

cockroach/pkg/sql/colflow/stats.go

Lines 97 to 99 in b05bab3

    
           func (bic *batchInfoCollector) next() { 
        
           	bic.batch = bic.Operator.Next() 
        
           }

in pkg/sql/colflow.(*batchInfoCollector).next

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/stats.go

Lines 105 to 107 in b05bab3

    
           // execution time (e.g. in the statement bundle). 
        
           err := colexecerror.CatchVectorizedRuntimeError(bic.next) 
        
           bic.stopwatch.Stop()

in pkg/sql/colflow.(*batchInfoCollector).Next

cockroach/pkg/sql/colexec/materializer.go

Lines 260 to 262 in b05bab3

    
           // Get a fresh batch. 
        
           m.batch = m.input.Next() 
        
           if m.batch.Length() == 0 {

in pkg/sql/colexec.(*Materializer).next

cockroach/pkg/sql/colexec/materializer.go

Lines 285 to 287 in b05bab3

    
           func (m *Materializer) nextAdapter() { 
        
           	m.outputRow = m.next() 
        
           }

in pkg/sql/colexec.(*Materializer).nextAdapter

cockroach/pkg/sql/colexecerror/error.go

Lines 90 to 92 in b05bab3

    
           }() 
        
           operation() 
        
           return retErr

in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colexec/materializer.go

Lines 291 to 293 in b05bab3

    
           for m.State == execinfra.StateRunning { 
        
           	if err := colexecerror.CatchVectorizedRuntimeError(m.nextAdapter); err != nil { 
        
           		m.MoveToDraining(err)

in pkg/sql/colexec.(*Materializer).Next

cockroach/pkg/sql/rowexec/stats.go

Lines 66 to 68 in b05bab3

    
           start := timeutil.Now() 
        
           row, meta := isc.RowSource.Next() 
        
           if row != nil {

in pkg/sql/rowexec.(*inputStatCollector).Next

cockroach/pkg/sql/rowexec/joinreader.go

Lines 663 to 665 in b05bab3

    
           for jr.curBatchSizeBytes < jr.batchSizeBytes { 
        
           	row, meta := jr.input.Next() 
        
           	if meta != nil {

in pkg/sql/rowexec.(*joinReader).readInput

cockroach/pkg/sql/rowexec/joinreader.go

Lines 602 to 604 in b05bab3

    
           case jrReadingInput: 
        
           	jr.runningState, row, meta = jr.readInput() 
        
           case jrPerformingLookup:

in pkg/sql/rowexec.(*joinReader).Next

cockroach/pkg/sql/colexec/columnarizer.go

Lines 221 to 223 in b05bab3

    
           for ; nRows < c.batch.Capacity(); nRows++ { 
        
           	row, meta := c.input.Next() 
        
           	if meta != nil {

in pkg/sql/colexec.(*Columnarizer).Next

pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 88
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 965
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 192
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Next at line 353
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colexec/colexecjoin/hashjoiner.go in pkg/sql/colexec/colexecjoin.(*hashJoiner).exec at line 474
pkg/sql/colexec/colexecjoin/hashjoiner.go in pkg/sql/colexec/colexecjoin.(*hashJoiner).Next at line 290
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next.func1 at line 198
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next at line 196
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*bufferExportingOperator).Next at line 320
pkg/sql/colexec/external_sort.go in pkg/sql/colexec.(*inputPartitioningOperator).Next at line 747
pkg/sql/colexec/sort.go in pkg/sql/colexec.(*allSpooler).spool at line 146
pkg/sql/colexec/sort.go in pkg/sql/colexec.(*sortOp).Next at line 283
pkg/sql/colexec/external_sort.go in pkg/sql/colexec.(*externalSorter).Next at line 347
pkg/sql/colexec/disk_spiller.go in pkg/sql/colexec.(*diskSpillerBase).Next at line 211
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 98
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 106
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 261
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 286
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 91
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 292
pkg/sql/rowexec/stats.go in pkg/sql/rowexec.(*inputStatCollector).Next at line 67
pkg/sql/rowexec/joinreader.go in pkg/sql/rowexec.(*joinReader).readInput at line 664
pkg/sql/rowexec/joinreader.go in pkg/sql/rowexec.(*joinReader).Next at line 603
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Next at line 222

Tag	Value
Cockroach Release	`v21.2.11`
Cockroach SHA:	`b05bab3`
Platform	linux amd64
Distribution	CCL
Environment	v21.2.11
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-17090

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-08-15T16:53:23Z

dup of #82947

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jun 28, 2022

yuzefovich closed this as completed Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) Wraps: (4) Wraps: (5) assertion failure Wraps: (6) attached stack trace -- stack trace: | githu... #83481

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) Wraps: (4) Wraps: (5) assertion failure Wraps: (6) attached stack trace -- stack trace: | githu... #83481

cockroach-teamcity commented Jun 28, 2022 •

edited by cockroach-jira-scripts

Loading

yuzefovich commented Aug 15, 2022

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) Wraps: (4) Wraps: (5) assertion failure Wraps: (6) attached stack trace -- stack trace: | githu... #83481

sentry: error.go:88: unexpected error from the vectorized engine: context canceled (1) Wraps: (2) Wraps: (3) Wraps: (4) Wraps: (5) assertion failure Wraps: (6) attached stack trace -- stack trace: | githu... #83481

Comments

cockroach-teamcity commented Jun 28, 2022 • edited by cockroach-jira-scripts Loading

yuzefovich commented Aug 15, 2022

cockroach-teamcity commented Jun 28, 2022 •

edited by cockroach-jira-scripts

Loading