Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

colexec: v20.1.3: unknown type error from cfetcher #56225

Closed
cockroach-teamcity opened this issue Nov 3, 2020 · 2 comments · Fixed by #56528
Closed

colexec: v20.1.3: unknown type error from cfetcher #56225

cockroach-teamcity opened this issue Nov 3, 2020 · 2 comments · Fixed by #56528
Assignees
@cathymw
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/2003521025/?referrer=webhooks_plugin

Panic message:

*barriers.barrierError: -- details for github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/withstack/*withstack.withStack::: (1)
*errutil.withMessage
*safedetails.withSafeDetails: unexpected error from the vectorized runtime (2)
error.go:92: *withstack.withStack (3)
*assert.withAssertionFailure
*contexttags.withContext: n<*base.NodeIDContainer> (4)
*contexttags.withContext: n<*base.NodeIDContainer> (5)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/colexec/execerror/error.go

Lines 91 to 93 in 7fd454f

// unexpected.
retErr = errors.NewAssertionErrorWithWrappedErrf(err, "unexpected error from the vectorized runtime")
}
in pkg/sql/colexec/execerror.CatchVectorizedRuntimeError.func1
/usr/local/go/src/runtime/panic.go#L678-L680 in runtime.gopanic

cockroach/pkg/sql/colexec/execerror/error.go

Lines 191 to 193 in 7fd454f

func VectorizedInternalPanic(err interface{}) {
panic(err)
}
in pkg/sql/colexec/execerror.VectorizedInternalPanic

cockroach/pkg/sql/colexec/colbatch_scan.go

Lines 69 to 71 in 7fd454f

if err != nil {
execerror.VectorizedInternalPanic(err)
}
in pkg/sql/colexec.(*colBatchScan).Next

cockroach/pkg/sql/colexec/cancel_checker.go

Lines 46 to 48 in 7fd454f

c.checkEveryCall(ctx)
return c.input.Next(ctx)
}
in pkg/sql/colexec.(*CancelChecker).Next
https://github.com/cockroachdb/cockroach/blob/7fd454f880f386cdd0eda6b21b12f6532c14f0db/pkg/sql/colexec/selection_ops.eg.go#L54421-L54423 in pkg/sql/colexec.(*selGETimestampTimestampConstOp).Next
https://github.com/cockroachdb/cockroach/blob/7fd454f880f386cdd0eda6b21b12f6532c14f0db/pkg/sql/colexec/selection_ops.eg.go#L53533-L53535 in pkg/sql/colexec.(*selLTTimestampTimestampConstOp).Next
https://github.com/cockroachdb/cockroach/blob/7fd454f880f386cdd0eda6b21b12f6532c14f0db/pkg/sql/colexec/select_in.eg.go#L577-L579 in pkg/sql/colexec.(*selectInOpBytes).Next

cockroach/pkg/sql/colexec/simple_project.go

Lines 120 to 122 in 7fd454f

func (d *simpleProjectOp) Next(ctx context.Context) coldata.Batch {
batch := d.input.Next(ctx)
projBatch, found := d.batches[batch]
in pkg/sql/colexec.(*simpleProjectOp).Next

cockroach/pkg/sql/colexec/fn_op.go

Lines 34 to 36 in 7fd454f

func (f fnOp) Next(ctx context.Context) coldata.Batch {
batch := f.input.Next(ctx)
f.fn()
in pkg/sql/colexec.fnOp.Next

cockroach/pkg/sql/colexec/one_shot.go

Lines 36 to 38 in 7fd454f

func (o *oneShotOp) Next(ctx context.Context) coldata.Batch {
batch := o.input.Next(ctx)
in pkg/sql/colexec.(*oneShotOp).Next

cockroach/pkg/sql/colexec/aggregator.go

Lines 390 to 392 in 7fd454f

for a.scratch.resumeIdx < a.scratch.outputSize {
batch := a.input.Next(ctx)
a.seenNonEmptyBatch = a.seenNonEmptyBatch || batch.Length() > 0
in pkg/sql/colexec.(*orderedAggregator).Next

cockroach/pkg/sql/colexec/deselector.go

Lines 50 to 52 in 7fd454f

p.resetOutput()
batch := p.input.Next(ctx)
if batch.Selection() == nil {
in pkg/sql/colexec.(*deselectorOp).Next

cockroach/pkg/sql/colflow/colrpc/outbox.go

Lines 242 to 244 in 7fd454f

}
o.batch = o.Input().Next(o.runnerCtx)
}
in pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1

cockroach/pkg/sql/colexec/execerror/error.go

Lines 94 to 96 in 7fd454f

}()
operation()
return retErr
in pkg/sql/colexec/execerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/colrpc/outbox.go

Lines 259 to 261 in 7fd454f

if err := execerror.CatchVectorizedRuntimeError(nextBatch); err != nil {
if log.V(1) {
in pkg/sql/colflow/colrpc.(*Outbox).sendBatches

cockroach/pkg/sql/colflow/colrpc/outbox.go

Lines 335 to 337 in 7fd454f

terminatedGracefully, errToSend := o.sendBatches(ctx, stream, cancelFn)
if terminatedGracefully || errToSend != nil {
in pkg/sql/colflow/colrpc.(*Outbox).runWithStream

cockroach/pkg/sql/colflow/colrpc/outbox.go

Lines 189 to 191 in 7fd454f

log.VEvent(ctx, 2, "Outbox starting normal operation")
o.runWithStream(ctx, stream, cancelFn)
log.VEvent(ctx, 2, "Outbox exiting")
in pkg/sql/colflow/colrpc.(*Outbox).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 611 to 613 in 7fd454f

run := func(ctx context.Context, cancelFn context.CancelFunc) {
outbox.Run(ctx, s.nodeDialer, stream.TargetNodeID, s.flowID, stream.StreamID, cancelFn)
currentOutboxes := atomic.AddInt32(&s.numOutboxes, -1)
in pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 1172 to 1174 in 7fd454f

go func() {
run(ctx, cancelFn)
if wg != nil {
in pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1
/usr/local/go/src/runtime/asm_amd64.s#L1356-L1358 in runtime.goexit

pkg/sql/colexec/execerror/error.go in pkg/sql/colexec/execerror.CatchVectorizedRuntimeError.func1 at line 92
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 679
pkg/sql/colexec/execerror/error.go in pkg/sql/colexec/execerror.VectorizedInternalPanic at line 192
pkg/sql/colexec/colbatch_scan.go in pkg/sql/colexec.(*colBatchScan).Next at line 70
pkg/sql/colexec/cancel_checker.go in pkg/sql/colexec.(*CancelChecker).Next at line 47
pkg/sql/colexec/selection_ops.eg.go in pkg/sql/colexec.(*selGETimestampTimestampConstOp).Next at line 54422
pkg/sql/colexec/selection_ops.eg.go in pkg/sql/colexec.(*selLTTimestampTimestampConstOp).Next at line 53534
pkg/sql/colexec/select_in.eg.go in pkg/sql/colexec.(*selectInOpBytes).Next at line 578
pkg/sql/colexec/simple_project.go in pkg/sql/colexec.(*simpleProjectOp).Next at line 121
pkg/sql/colexec/fn_op.go in pkg/sql/colexec.fnOp.Next at line 35
pkg/sql/colexec/one_shot.go in pkg/sql/colexec.(*oneShotOp).Next at line 37
pkg/sql/colexec/aggregator.go in pkg/sql/colexec.(*orderedAggregator).Next at line 391
pkg/sql/colexec/deselector.go in pkg/sql/colexec.(*deselectorOp).Next at line 51
pkg/sql/colflow/colrpc/outbox.go in pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1 at line 243
pkg/sql/colexec/execerror/error.go in pkg/sql/colexec/execerror.CatchVectorizedRuntimeError at line 95
pkg/sql/colflow/colrpc/outbox.go in pkg/sql/colflow/colrpc.(*Outbox).sendBatches at line 260
pkg/sql/colflow/colrpc/outbox.go in pkg/sql/colflow/colrpc.(*Outbox).runWithStream at line 336
pkg/sql/colflow/colrpc/outbox.go in pkg/sql/colflow/colrpc.(*Outbox).Run at line 190
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1 at line 612
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1 at line 1173
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1357
Tag Value
Cockroach Release v20.1.3
Cockroach SHA: 7fd454f
Platform linux amd64
Distribution CCL
Environment v20.1.3
Command server
Go Version go1.13.9
# of CPUs 16
# of Goroutines 1489
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Nov 3, 2020
@yuzefovich
Copy link
Member 

-- details for github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/withstack/*withstack.withStack:::

github.com/cockroachdb/cockroach/pkg/util/encoding.PeekValueLengthWithOffsetsAndType
	/go/src/github.com/cockroachdb/cockroach/pkg/util/encoding/encoding.go:2403
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).processValueBytes
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:1157
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).processValueTuple
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:1209
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).processValue
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:958
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextBatch
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:749
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextAdapter
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:597
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Allocator).PerformOperation
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/allocator.go:220
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).NextBatch
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:589
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*colBatchScan).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colbatch_scan.go:68
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*CancelChecker).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cancel_checker.go:47
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*selGETimestampTimestampConstOp).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/selection_ops.eg.go:54422
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*selLTTimestampTimestampConstOp).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/selection_ops.eg.go:53534
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*selectInOpBytes).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/select_in.eg.go:578
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*simpleProjectOp).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/simple_project.go:121
github.com/cockroachdb/cockroach/pkg/sql/colexec.fnOp.Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/fn_op.go:35
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*oneShotOp).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/one_shot.go:37
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*orderedAggregator).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/aggregator.go:391
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*deselectorOp).Next
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/deselector.go:51
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:243
github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror.CatchVectorizedRuntimeError
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror/error.go:95
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:260
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).runWithStream
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:336
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:190
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:612
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1173
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357
-- details for github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/safedetails/*safedetails.withSafeDetails:::
unknown type %s
-- arg 1: <encoding.Type>
-- details for errors/*errors.errorString:::

@yuzefovich yuzefovich changed the title sentry: *barriers.barrierError: -- details for github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/withstack/*withstack.withStack::: (1) *errutil.withMessage *safedetails.withSafeDetails: unexpected error from the vectorized runtime (2) error.go:92: *withstack.withStack (3) *assert.withAssertionFailure *contexttags.withContext: n<*base.NodeIDContainer> (4) *contexttags.withContext: n<*base.NodeIDContainer> (5) (check the extra data payloads) colexec: v20.1.3: unknown type error from cfetcher Nov 3, 2020
@asubiotto
Copy link
Contributor

What we need to do here is mark the type safe to be printed (search for Redactable) and try to figure out which type isn't handled.

@cathymw cathymw mentioned this issue Nov 10, 2020
Merged
@craig craig bot closed this as completed in 39a5bb6 Nov 19, 2020
@craig craig bot closed this as completed in #56528 Nov 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cathymw cathymw

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

colexec: mark encoding type as safe to print cathymw/cockroach
4 participants
@yuzefovich @asubiotto @cockroach-teamcity @cathymw