sql,changefeedccl: miscellaneous cleanup of server shutdown #125357
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, we would use "vanilla" goroutines when refreshing stats entries (triggered by the rangefeed), but this has a downside of not respecting the stopper signal to shutdown. That could cause the "short-living monitors are not stopped" assertion to fire on the server shutdown in tests. This commit prevents that by using the stopper to start async tasks for the entry refresh (which has additional benefits like recovering from panics). Release note: None
This commit is similar in spirit as the previous one and replaces usage of the "vanilla" goroutine with an async task connected to the stopper for the hookFn plan node. This allows us to synchronize the server shutdown and cancellation of the hook task. Release note: None
|
This change is |
yuzefovich
force-pushed
the
fix-assertion
branch
2 times, most recently
from
2c8d00a to
cfb3731
Compare
This commit introduces a "root" changefeed memory monitor that will be the parent for all CDC DistSQL flows. The primary motivation behind this change is to disconnect the CDC flow from the planner's monitor (which is "txn" monitor of the connection that started the changefeed). This is needed since that connection can be closed _before_ the CDC flow is cleaned up which triggers the "short-living non-stopped monitor" assertion. An additional benefit of this change is improved observability since we can now easily track the memory usage across all changefeeds. Release note: None
yuzefovich
requested review from
msbutler
and removed request for
a team and
msbutler
rharding6373
approved these changes
Jun 10, 2024
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @michae2)
|
TFTR! bors r+ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
stats: refresh stats cache entries in async tasks
Previously, we would use "vanilla" goroutines when refreshing stats entries (triggered by the rangefeed), but this has a downside of not respecting the stopper signal to shutdown. That could cause the "short-living monitors are not stopped" assertion to fire on the server shutdown in tests. This commit prevents that by using the stopper to start async tasks for the entry refresh (which has additional benefits like recovering from panics).
Fixes: #125329.
sql: use async task for plan hook goroutine
This commit is similar in spirit as the previous one and replaces usage of the "vanilla" goroutine with an async task connected to the stopper for the hookFn plan node. This allows us to synchronize the server shutdown and cancellation of the hook task.
Fixes: #125139.
changefeedccl: create separate root changefeed memory monitor
This commit introduces a "root" changefeed memory monitor that will be the parent for all CDC DistSQL flows. The primary motivation behind this change is to disconnect the CDC flow from the planner's monitor (which is "txn" monitor of the connection that started the changefeed). This is needed since that connection can be closed before the CDC flow is cleaned up which triggers the "short-living non-stopped monitor" assertion. An additional benefit of this change is improved observability since we can now easily track the memory usage across all changefeeds.
Release note: None