Reload database on SIGHUP #2422
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hugoghx
force-pushed
the
hugoamvieira-controller-dbswap
branch
2 times, most recently
from
4c26993 to
1ebeb0e
Compare
jefferai
force-pushed
the
hugoamvieira-controller-dbswap
branch
2 times, most recently
from
4b9cbe4 to
e5db129
Compare
hugoghx
force-pushed
the
hugoamvieira-controller-dbswap
branch
2 times, most recently
from
bca8007 to
34d500a
Compare
jefferai
approved these changes
Sep 12, 2022
This commit introduces closing the Schema Manager's underlying driver connection. We were only releasing the shared lock but not actually cleaning up the db connection and relied on the fact that the process would exit to do that for us.
This commit extracts 3 separate pieces of functionality into their own function/method: - Extracts the database opening with the intention to have a function that performs all that functionality but returns the created database object rather than setting it to the Server structure. - Extracts database validation logic, to provide a common code path for upcoming functionality. - Expose schema manager on the Command struct as well as providing common functions to acquire shared lock/close.
If the database url is updated on Boundary's config and then a SIGHUP triggered, we now connect to the new database, ensure it's OK to use and use that one going forwards instead. This allows a controller to be deployed in a more dynamic infrastructure where, for example, database credentials could change, without having to restart the controller. In this case only a database url config change and a SIGHUP signal to all controllers would be required. This uses an atomic.Pointer to store wrapped DB objects. This allows atomically swapping to the new object. A timeout is introduced so that if the object is swapped after the existing one has been looked up it isn't immediately invalidated. The timeout is a maximum; a ticker once a second that looks at stats to determine how many database connections are still in use and then closes the driver as soon as none are in use. This could be done with mutexes to avoid checking stats in a loop, but that carries its own risk: if a write lock is attempted to be acquired while a query is holding a read lock, the write lock will block, and any subsequent read locks will also block since write locks take priority. As a result, if a query is long-running or timing out, all other queries in the system will become blocked. Our default user request timeout is 90 seconds and some jobs have no timeout at all, so this is not a purely theoretical concern, even if it's unlikely. Refs: #2435 Co-authored-by: Jeff Mitchell <[email protected]>
hugoghx
force-pushed
the
hugoamvieira-controller-dbswap
branch
from
34d500a to
9002930
Compare
tmessi
pushed a commit
that referenced
this pull request
Sep 12, 2022
If the database url is updated on Boundary's config and then a SIGHUP triggered, we now connect to the new database, ensure it's OK to use and use that one going forwards instead. This allows a controller to be deployed in a more dynamic infrastructure where, for example, database credentials could change, without having to restart the controller. This operation is done atomically to prevent errors and inconsistency. A timeout is present so that if the object is swapped after the existing one has been looked up it isn't immediately invalidated. The timeout is a maximum; a ticker once a second that looks at stats to determine how many database connections are still in use and then closes the driver as soon as none are in use.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the database url is updated on Boundary's config and then a SIGHUP
triggered, we now connect to the new database, ensure it's OK to use
and use that one going forwards instead.