Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(shutdown): stop all running jobs before stopping workflow (#423) #423

Merged
merged 4 commits into from
Feb 1, 2024

Conversation

mdonadoni
Copy link
Member

@mdonadoni mdonadoni commented Jan 23, 2024

Closes reanahub/reana-workflow-controller#546

mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this pull request Jan 23, 2024
…hub#423)

Make sure that all the running jobs are stopped before stopping the
run-batch pod, as otherwise they will continue running and they will not
be cleaned up.

Closes reanahub/reana-workflow-controller#546
Copy link

codecov bot commented Jan 23, 2024

Codecov Report

Attention: 106 lines in your changes are missing coverage. Please review.

Comparison is base (8a2757e) 47.78% compared to head (99c097e) 46.46%.

❗ Current head 99c097e differs from pull request most recent head 866675b. Consider uploading reports for the commit 866675b to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #423      +/-   ##
==========================================
- Coverage   47.78%   46.46%   -1.32%     
==========================================
  Files          17       17              
  Lines        1149     1231      +82     
==========================================
+ Hits          549      572      +23     
- Misses        600      659      +59     
Files Coverage Δ
reana_job_controller/job_manager.py 89.65% <100.00%> (+0.18%) ⬆️
reana_job_controller/slurmcern_job_manager.py 25.71% <40.00%> (-0.03%) ⬇️
reana_job_controller/htcondorcern_job_manager.py 0.00% <0.00%> (ø)
reana_job_controller/job_db.py 34.69% <21.05%> (-10.47%) ⬇️
reana_job_controller/job_monitor.py 48.03% <10.00%> (+1.10%) ⬆️
reana_job_controller/kubernetes_job_manager.py 62.12% <51.06%> (-2.96%) ⬇️
reana_job_controller/rest.py 40.76% <26.31%> (-8.24%) ⬇️

Copy link
Member

@giuseppe-steduto giuseppe-steduto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well and looks good to me! I personally don't have any special comments :)

I was not sure about the shutdown scope for the commit message, but it's more or less user-facing, it describes the affected functionality of r-j-controller, and I cannot really think of a better alternative

@giuseppe-steduto
Copy link
Member

I cannot really think of a better alternative

Maybe progress is a valid one?

mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this pull request Jan 30, 2024
mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this pull request Jan 30, 2024
)

Up until now, reana-workflow-controller was updating the status of jobs
in the database. The updates are now managed by reana-job-controller
instead, as reana-workflow-controller should not be concerned by this.
mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this pull request Jan 30, 2024
mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this pull request Jan 30, 2024
…hub#423)

Make sure that all the running jobs are stopped before stopping the
run-batch pod, as otherwise they will continue running and they will not
be cleaned up.

Closes reanahub/reana-workflow-controller#546
Up until now, reana-workflow-controller was updating the status of jobs
in the database. The updates are now managed by reana-job-controller
instead, as reana-workflow-controller should not be concerned by this.
…hub#423)

Make sure that all the running jobs are stopped before stopping the
run-batch pod, as otherwise they will continue running and they will not
be cleaned up.

Closes reanahub/reana-workflow-controller#546
@mdonadoni mdonadoni merged commit 866675b into reanahub:master Feb 1, 2024
12 checks passed
@mdonadoni mdonadoni deleted the fix-runaway-pods branch February 1, 2024 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

job pods keep running even after workflow failure
2 participants