Deal with stale daemon PID files during stop attempts #2795
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@giovannipizzi , requested your review. Even though you co-authored the commit and the idea for the implementation was yours, you didn't touch any of the real code in the end so maybe it is okay for you to review. |
ConradJohnston
force-pushed
the
fix_2775
branch
from
9a51e4e
to
6b42ad7
Compare
ConradJohnston
force-pushed
the
fix_2775
branch
2 times, most recently
from
fcfc71e
to
4ea906c
Compare
|
What do we want to do get this over the line? Remove the dependence on psutil? I think this is the only place it's used in AiiDA now. |
In some circumstances, the circus PID file which stores the PID of the running circus process, can be corrupted or not deleted properly, causing calls to `is_daemon_running` to return a false positive. This situation can arise if a system is shut down suddenly and so the process is killed but the PID file is not deleted in time. Alternatively, another process or the user may have meddled with the PID file in some way, corrupting it. Here, we implement a function that checks for stale PID files by checking whether the PID contained in the PID file matches a valid running `verdi `process. If it does not, a warning is emitted and the PID file is deleted. This function is called when `verdi daemon stop` is called, making it function as a hard reset in a situation of a stale PID. Co-authored-by: Giovanni Pizzi <[email protected]>
sphuber
changed the title
|
For now, I think we can leave the |
sphuber
approved these changes
May 30, 2019
This was referenced May 30, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2775 and fixes #1865
In some circumstances, the circus PID file which stores the PID of the running circus process can be corrupted, causing calls to the
get_daemon_pid()method to return the wrong PID, or causing theis_daemon_runningattribute to returnTruewhen in reality the daemon is not running.This function checks if the PID contained in the circus pid file (
circus-{PROFILE_NAME}.pid) matches a valid runningverdiprocess. If it does not, the PID file is stale and will be removed.This situation can arise if a system is shut down suddenly and so the process is killed but the PID file is not deleted in time. Alternatively, another process or the user may have meddled with the PID file in some way, corrupting it.
Example of broken behaviour:
verdi daemon startpid=$(cat ~/.aiida/daemon/circus-{PROFILE_NAME}.pid)echo '9999999999' > ~/.aiida/daemon/circus-{PROFILE_NAME}.pidkill $pidverdi daemon stopIn this scenario, the
daemon stopcommand will timeout, as will thestartcommand, due to the stale PID file which is lying about there being a running daemon with the claimed PID.