Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colexec: fix span use after finish in edge cases in parallel sync #84873

Merged
merged 1 commit into from
Jul 22, 2022

Conversation

yuzefovich
Copy link
Member

@yuzefovich yuzefovich commented Jul 21, 2022

Previously, in some cases the parallel unordered synchronizer would
finish the tracing spans prematurely which would result in "use of Span
after Finish" panics in test builds and this is now fixed.

Release note: None

Previously, in some cases the parallel unordered synchronizer would
finish the tracing spans prematurely which would result "use of Span
after Finish" panics in test builds and this is now fixed.

Release note: None
@yuzefovich yuzefovich requested review from michae2, cucaroach and a team July 21, 2022 20:25
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@yuzefovich
Copy link
Member Author

Saw this panic on one of the CI runs:

panic: use of Span after Finish. Span: parallel unordered sync input 1. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: parallel unordered sync input 1. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: parallel unordered sync input 1. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

goroutine 2452008 [running]:
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Wait(0xc04901a960)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:506 +0x1d9
panic({0x47de820, 0xc000840e70})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:58 +0x3bb
panic({0x47de820, 0xc000840e70})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x9ab6dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:186 +0x134
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc00fdeae00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:411 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:419
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x68cdc68, 0xc042235200})
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x68cdc68, 0xc042235200}, 0x0, 0x1, 0x26b27950, {0x50e4c67, 0x12}, {0x0, 0x0, 0x0})
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:206 +0x106
github.com/cockroachdb/cockroach/pkg/util/log.VEventf(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:232
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*orderedAggregator).close(0xc045f48c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:375 +0x67
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*orderedAggregator).ConsumerClosed(0xc02fad6e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/aggregator.go:738 +0x19
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx(0xc0154fdc00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:915 +0x159
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalClose(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:903
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Close(0xc0c533, {0x68d0be0, 0xc0154fdc00})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:295 +0x33
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Close(0x0, {0x68cdc68, 0xc042234d20})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:128 +0x4f
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupFlow.func1.1.1({0x68cdc68, 0xc042234d20})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1208 +0x68
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*callbackCloser).Close(0xc0422355c0, {0x68cdc68, 0xc042234d20})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1116 +0x2c
github.com/cockroachdb/cockroach/pkg/sql/colexecop.Closers.Close({0xc01c9f2c60, 0x1, 0xc008fa6d68}, {0x68cdc68, 0xc042234d20})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:190 +0x99
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Close(0xc019809320, {0x68cdc68, 0xc042234d20})
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:497 +0x1df

Copy link
Collaborator

@michae2 michae2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @cucaroach)

@yuzefovich
Copy link
Member Author

TFTR!

bors r+

@craig
Copy link
Contributor

craig bot commented Jul 22, 2022

Build succeeded:

@craig craig bot merged commit 02727e3 into cockroachdb:master Jul 22, 2022
@yuzefovich yuzefovich deleted the parallel-sync-fix branch July 22, 2022 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants