Catching create job failures, updates to job definition

[3coins]

• Current task runner will end the async loop if any exceptions are raised by the scheduler while creating the job. Fixed this by catching any exceptions during create_job step.
• Resuming job definition after pausing will not add the task to the queue in certain scenarios. Fixed this by ensuring that on job definition update, task is added to the queue if not present. To reproduce
  1. Create a scheduled job that runs every 2 mins, let the first job run.
  2. Pause the job definition
  3. Resume the job definition within the next 2 mins
  4. Job will not run at the end of 2 mins

[github-actions]

👈 Launch a Binder on branch 3coins/jupyter-scheduler/catch-exceptions

[dlqqq]

Ah I see the issue with the current logic. Let's say creates a job definition to run every minute, and suppose the following actions occur:

t	action
0.0	job definition created
0.3	job definition paused => cache.next_run_time = 1
0.3 + poll_interval	process_queue() executes, removes task from heap because active == False
0.7	job definition resume => next_run_time == 1 and next_run_time == cache.next_run_time => task is not added back into the heap

To fix this, I don't think we need to scan the entire heap when updating a job definition. Instead we just need to change the condition to:

new_next_run_time = cached_next_run_time != next_run_time and active
resumed = model.active and not cache.active

... # update cache

if (new_next_run_time or resumed):
  ... # add task to heap

[3coins]

@dlqqq Updated as per our discussion.