-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
colexec: fix recent regression with cancellation of inboxes #79716
Conversation
343776a
to
bb91ffe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @michae2)
Recent change 773d9ca fixed the way inboxes handle regular query errors so that now the gRPC streams are not broken whenever a query error is encountered. However, that change introduced a regression - it is now possible that the inbox handler goroutine (the one instantiated to handle FlowStream gRPC call) never exits when the inbox is an input to the parallel unordered synchronizer. In particular, the following sequence of events can happen: 1. the reader goroutine of the inbox receives an error from the corresponding outbox 2. this error is propagated to one of the input goroutines of the unordered synchronizer via a panic. Notably, this is considered a "graceful" termination from the perspective of the gRPC stream handling, so the handler goroutine is not notified of this error, and the inbox is not closed. It is expected that the inbox will be drained which will close the handler goroutine. 3. however, the synchronizer input goroutine currently simply receives the error, propagates it to the coordinator goroutine, and exits, without performing the draining. Thus, we get into such a state that the inbox is never drained, so the handler goroutine will stay alive forever. In particular, this will block `Flow.Wait` calls, and `Flow.Cleanup` will never be called. This could leak, for example, to leaking file descriptors used by the temporary disk storage in the vectorized engine. This fix is quite simple - instead of exiting in step 3, the synchronizer's input goroutine should just proceed to draining, and only exit once draining is performed. I believe this was always the intention, but it didn't really matter until the fix in 773d9ca because draining of the inbox was a noop since the gRPC stream was prematurely broken. Release note: None
bb91ffe
to
ecd6403
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed a typo in the comment.
TFTR!
bors r+
Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @cucaroach and @michae2)
Build succeeded: |
Recent change 773d9ca fixed the way
inboxes handle regular query errors so that now the gRPC streams are not
broken whenever a query error is encountered. However, that change
introduced a regression - it is now possible that the inbox handler
goroutine (the one instantiated to handle FlowStream gRPC call) never
exits when the inbox is an input to the parallel unordered synchronizer.
In particular, the following sequence of events can happen:
corresponding outbox
unordered synchronizer via a panic. Notably, this is considered
a "graceful" termination from the perspective of the gRPC stream
handling, so the handler goroutine is not notified of this error, and
the inbox is not closed. It is expected that the inbox will be drained
which will close the handler goroutine.
the error, propagates it to the coordinator goroutine, and exits,
without performing the draining.
Thus, we get into such a state that the inbox is never drained, so the
handler goroutine will stay alive forever. In particular, this will
block
Flow.Wait
calls, andFlow.Cleanup
will never be called. Thiscould leak, for example, to leaking file descriptors used by the
temporary disk storage in the vectorized engine.
This fix is quite simple - instead of exiting in step 3, the
synchronizer's input goroutine should just proceed to draining, and only
exit once draining is performed. I believe this was always the
intention, but it didn't really matter until the fix in
773d9ca because draining of the inbox
was a noop since the gRPC stream was prematurely broken.
Fixes: #79469.
Release note: None