Understand and support use-cases for in-place notebooks edits

[wjchulme]

Due to the way the job-runner manages files, it's not currently possible to execute and save a .ipynb notebook, unless you save it as something else (either a different name, file location, or file type).

So you either have to save it to a different location and then run another action to put it back (using file system operations in R, say), or execute manually on the server.

[HelenCEBM]

I think this will be useful for various updating reports because currently, making a small change in the text that doesn't affect the results requires many steps (update locally, push to branch, merge, submit to job runner, wait, log in to server, copy and check files for release, push to new branch, merge again). It would be great if this kind of small change could just be the first 3 of these steps!

[bloodearnest]

Trying a manual tag: @brianmackenna

[iaindillingham]

Summary of the issue:

• For an in-place edit, the notebook file must be copied into and out of the container.
• For the notebook file to be copied out of the container, it must be listed in outputs.
• However, if the notebook file is listed in outputs, then it is not copied into the container.

A workaround is to save the edited notebook file to a new location, which is listed in outputs, and then overwrite the original notebook file with the edited notebook file in a subsequent action.

[evansd]

We're going to have to come up with an alternative workflow here as all the workarounds are essentially just exploiting bugs in job-runner which will fix at some point.

I think what we need to do is:

• collect a series of use-cases where people want to be able to do in-place updates, and their reasons for wanting to do so;
• work out the "correct" workflow for handling these;
• create the documentation and/or tooling needed to make these workflows easy for users.

[benbc]

Closing in favour of this option in our pipeline.