Features wish list (from Sangeeta)

[richfitz]

Manually migrating issue vimc/orderly2#1 from @sangeetabhatia03 after archiving that repo

---

My wish list for orderly2 (as much as I love orderly).

• Easily generating large orderly.yml – I would like to be able to generate orderly.yml on the fly using some orderly helper functions. E.g., a common use case is to depend on all tasks where parameter 1 was 1, and parameter 2 was anything (this is an example you mention in the readme), or parameter 2 is in say (a,b,c). My current approach to this is have tons of R scripts writing orderly.yml - one for each task. But they are all doing the same thing and could go into some orderly helper package.
• An easy way of interacting with orderly database to find out things like – (i) for what set of parameters have I run a specific report, (ii) what days a report was run on, (iii) generate a list of packages used across all my tasks.
• Some utility to draw a full dependency tree. My orderly projects start small but before I realise it, the number of tasks explodes. It would be useful for the easy documentation if I could easily generate a complete list of what depends on what.
• Playing nice with a tool like renv. I have run into the common issue of outputs being dissimilar across computers because of differences in package versions. I am exploring the use of tools like renv and I am hoping that I can use them in conjunction with orderly. My goal here is to be able to indicate my orderly project specific set of packages, so that my orderly project is self-contained.
• Cleanly delete archived copies. I often wish to delete reports that I have archived because my code has changed and the archived versions are of no use to me. This was particularly an issue when the outputs were big and I had limited space on my machine. I think I can see why this would not be a good idea wrt reproducibility but one instance where the user should be able to delete them easily if they have been pulled from some remote?

Thanks again for orderly, it makes my life so much easier!

[richfitz]

Easily generating large orderly.yml – I would like to be able to generate orderly.yml on the fly using some orderly helper functions. E.g., a common use case is to depend on all tasks where parameter 1 was 1, and parameter 2 was anything (this is an example you mention in the readme), or parameter 2 is in say (a,b,c). My current approach to this is have tons of R scripts writing orderly.yml - one for each task. But they are all doing the same thing and could go into some orderly helper package.

This is now no longer necessary, as there is no more yaml. You can compute your dependencies now and this is documented here

An easy way of interacting with orderly database to find out things like – (i) for what set of parameters have I run a specific report, (ii) what days a report was run on, (iii) generate a list of packages used across all my tasks.

See https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html which does this; the package processing will take some more work

Some utility to draw a full dependency tree. My orderly projects start small but before I realise it, the number of tasks explodes. It would be useful for the easy documentation if I could easily generate a complete list of what depends on what.

This is something not yet done, partly because it's hard, partly because we've focused on implementing the previous features first. With computed dependencies, we may never be able to draw a good dependency tree between source tasks, but it's possible. For the final graph among packets we have some ideas but nothing yet that is nice to work with.

Playing nice with a tool like renv. I have run into the common issue of outputs being dissimilar across computers because of differences in package versions. I am exploring the use of tools like renv and I am hoping that I can use them in conjunction with orderly. My goal here is to be able to indicate my orderly project specific set of packages, so that my orderly project is self-contained.

We're exploring this with a different project and will let you know. Please let us know if you get anywhere yourself.

Cleanly delete archived copies. I often wish to delete reports that I have archived because my code has changed and the archived versions are of no use to me. This was particularly an issue when the outputs were big and I had limited space on my machine. I think I can see why this would not be a good idea wrt reproducibility but one instance where the user should be able to delete them easily if they have been pulled from some remote?

Delete whatever you want now!

[richfitz]

Cleanly delete archived copies. I often wish to delete reports that I have archived because my code has changed and the archived versions are of no use to me. This was particularly an issue when the outputs were big and I had limited space on my machine. I think I can see why this would not be a good idea wrt reproducibility but one instance where the user should be able to delete them easily if they have been pulled from some remote?

With #94, you can now delete whatever you want and orderly will be able to recover gracefully (obviously it can't recover information that was deleted so there are limits to grace here). This has been documented in the intro vignette https://mrc-ide.github.io/orderly2/articles/introduction.html#deleting-things-from-the-archive