Skip to content

Commit

Permalink
Raise error if the job is aborted
Browse files Browse the repository at this point in the history
I am not sure why the behaviour was changed. It is mission critical.

It is essential that the pipelines raise an error if the job is cancelled by
the scheduler. A warning is not enough. It is not reasonbale to expect people to check
the logs for warnings 100% of the time.

Jobs may be cancelled by the scheduler for exceeding memory limits and in such cases
this may result in incomplete output. Downstream tasks may still run, and the
incomplete output may go undetected but might change the outputs in a meaningful way,
which ultimately may lead to incorrect interpretation of data.
  • Loading branch information
snsansom committed Jun 14, 2020
1 parent e09683b commit 2ccd52a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion cgatcore/pipeline/cluster.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ def collect_single_job_from_cluster(self,
error_msg = None
if retval.exitStatus == 0:
if retval.wasAborted is True:
get_logger().warning(
error_msg = (
"Job {} has exit status 0, but marked as hasAborted=True, hasExited={} "
"(Job may have been cancelled by the user or the scheduler due to memory constraints)"
"The stderr was \n{}\nstatement = {}".format(
Expand Down

0 comments on commit 2ccd52a

Please sign in to comment.