Handle DWS fatal errors #44
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Set Flags=TeardownFailure in burst_buffer.conf. This tells Slurm to go to Teardown if there are errors during stage_in or stage_out. In the burst_buffer.lua plugin, adjust slurm_bb_job_teardown() to check the workflow for any fatal errors recorded in the drivers array. If any are found, then we know we were called in a fatal error situation and we not only delete the workflow, per usual, but we also call the Slurm `scancel` command to cancel the Slurm job. Signed-off-by: Dean Roehrich <[email protected]>
roehrich-hpe
requested review from
ajfloeder,
matthew-richerson and
bdevcich
bdevcich
approved these changes
Sep 22, 2023
|
Signed-off-by: Dean Roehrich <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Set Flags=TeardownFailure in burst_buffer.conf. This tells Slurm to go to Teardown if there are errors during stage_in or stage_out.
In the burst_buffer.lua plugin, adjust slurm_bb_job_teardown() to check the workflow for any fatal errors recorded in the drivers array. If any are found, then we know we were called in a fatal error situation and we not only delete the workflow, per usual, but we also call the Slurm
scancelcommand to cancel the Slurm job.