Enabled outputs for failed jobs

[3coins]

Summary

Failures for jobs can be of 2 types:

1. Failures while creating the job, so no outputs are created.
2. Failure while executing the notebooks due to an error within the notebook cell. These can generate outputs, but can still act as a failed job.

Current implementation doesn't allow for distinguishing between 1 and 2, so downloads for failed jobs that might have outputs are not enabled. This can lead to confusion by users, who don't have access to the failed notebook output or logs to understand where the problem existed.

Solution

• Marked jobs with cell execution error as failed, ensured output files are written for this case.
• Added error handling for downloader to ensure missing files doesn't block other file downloads.

[github-actions]

👈 Launch a Binder on branch 3coins/jupyter-scheduler/failure-outputs

[ellisonbg]

Can we avoid adding an extra status value by just allowing output files in the existing failure state?

[3coins]

@ellisonbg and I discussed the current approach at length, and came to a conclusion that the backend doesn't have a way to know which files are actually written after a job execution (especially on failure). The add_job_files and get_staging_paths always point to a full set of files, not a subset based on the job status. Due to this, an implicit contract in current backend has emerged that requires all job files to be present regardless of the failed or success status.

[dlqqq]

Jason already commented on frontend. The backend logic looks good.