Convert a recipe to a synda selection file

[bouweandela]

It would be great if we could have a command line option to convert a recipe to a synda selection file. The resulting selection file can then be used to install the data required to run the recipe with the command

synda install -s <selection-file>

[zklaus]

Synda selection files are really tricky. What works for CMIP5 might not work for CMIP6 and the handling of ensembles is awkward at best. Or at least it was. With everything going on with synda recently, I would double and triple check whether a particular formulation of the selection files is still the way to go.

[bouweandela]

Just created an issue to ask about it: ESPRI-Mod/synda#188

[bouweandela]

Another tool for downloading CMIP data is sproket, it uses a json configuration file. We could also generate that from the recipe.

[zklaus]

Sproket, though, is written in Go, which would add another set of dependencies. Last I checked it also did not support downloading many files in the ESGF directory structure, which would be the reason to use synda selection in the first place, right?

Anyway, I am not sure we need additional downloading capabilities at this point. Is there a user request for this? Otherwise, I would suggest addressing this after the 2.3.0 release and focussing on the milestone for now.

[bouweandela]

I see two use cases for this feature:

• Making it easy to request all the data you need for running a recipe from your system administrator if you are working on Jasmin/Mistral/MareNostrum/etc. The person in charge of downloading data at BSC uses sproket.
• Download all the data needed for running a recipe, useful if you want to run a recipe multiple times.

Because neither synda nor sproket can be installed from PyPI, I do not foresee adding either package as a dependency of the ESMValCore package. Note that installing sproket is pretty trivial though, it's a single executable file that you can download from their releases page.

[zklaus]

@bouweandela, how do you feel about this issue? Particularly in light of #1217, #1130, and #788. Do you still think this could be useful, or should we just close it? If we keep it open, do you want to target 2.4.0 or 2.5.0?

[bouweandela]

I would prefer to keep it open for now: I plan to clean up the datasets definition in the recipe parser as part of #1128, so hopefully this will be really easy to implement once that is done. I'll move it to 2.5 for now.

[bouweandela]

I did not manage to finish this before v2.5 and I'm not sure if I will have time to work on it in the future.

[bouweandela]

Synda has now been replaced by esgf-download.