forcibly close DB and cache socket connection post-fork #8580
+23
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryanpetrello
commented
Nov 12, 2020
| def __clean_on_fork__(self): | ||
| pid = os.getpid() | ||
| if pid not in self.__dict__['__forks__']: | ||
| self.__dict__['__forks__'][pid] = True |
There was a problem hiding this comment.
We have to access this via __dict__ instead of getattr to avoid a recursive loop (because __getattr__ calls this function via _get_local).
There was a problem hiding this comment.
I also tested this out by printing this dict in a tight loop of awx-python forks, and the data structure for each process never grows beyond two keys (the parent pid, and the current process pid).
So in effect, this branch is entered one time when a child process does getattr(settings, ...), and then after that, it's a no-op.
|
Build failed.
|
ryanpetrello
force-pushed
the
fork-race
branch
from
8ceb349 to
8317dd7
Compare
|
Build succeeded.
|
ryanpetrello
force-pushed
the
fork-race
branch
from
8317dd7 to
12a237f
Compare
we've seen evidence of a race condition on fork for awx.conf.Setting access; in the past, we've attempted to solve this by explicitly closing connections pre-fork, but we've seen evidence that this isn't always good enough this patch is an attempt to close connections post-fork so that sockets aren't inherited post fork, leading to bizarre race conditions in setting access
ryanpetrello
force-pushed
the
fork-race
branch
from
12a237f to
adfd8ed
Compare
|
Build succeeded.
|
matburt
approved these changes
Nov 19, 2020
|
Build succeeded (gate pipeline).
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
we've seen evidence of a race condition on fork for
awx.conf.model.Settingaccess; in the past, we've attempted to solve this by explicitly closing
connections pre-fork, but we've seen evidence that this isn't always
good enough
this patch is an attempt to close connections post-fork so that sockets
aren't inherited post fork, leading to bizarre race conditions in
setting access