Requirements re-haul

[bjlittle]

This PR rationalises the dependencies of iris for using conda with the CI travis-ci and readthedocs services, and also for PyPI.

Our current custom approach to manage both conda and PyPI dependencies as one, in general is not intuitive, nor standard, and leads to confusion for both developers and users alike.

Rather, this PR adopts standard community known approaches using self-contained conda environment yml files, and setup.py to manage PyPI dependencies for iris.

All the dependencies now live under the root directory requirements:

requirements
|-- all.txt
|-- ci
|   |-- iris.yml -> py37.yml
|   |-- py36.yml
|   |-- py37.yml
|   `-- readthedocs.yml -> iris.yml
|-- core.txt
|-- docs.txt
|-- setup.txt
`-- test.txt

For conda

As a developer, configuring iris with all the relevant dependencies that you need to code, test, and build the docs is now as simple as:

> conda env create --file=requirements/ci/iris.yml
> . activate iris-dev

Using conda over pip is preferred, due to the superior package dependency management and package coverage. Note that, for conda all packages are pulled from conda-forge and the associated yml files will create the named environment called iris-dev.

So it's one environment for all when it comes to coding, testing, and documentation (although now we can differentiate package content and package versions based on the Python version) which ensures (as far a possible without freezing the environment) that there is environment stability no matter what you are doing i.e., there are no package or package version differences for travis-ci, readthedocs, coding, local testing or building the docs.

The minor downside is that there is some duplication, but IMHO the simplicity and utility of this approach far outweighs the disadvantages of our current custom mechanisms to cherry-pick and manage package dependencies.

Note that, the readthedocs.yml is an alias for the iris.yml file, which is an alias for py37.yml file. As we roll forwards with testing iris with more modern versions of python, we simple create those Python version specific yml files e.g., py38.yml, re-point iris.yml to py38.py, and update .travis-ci.yml appropriately to update the test matrix.

For pip

It's not possible to only use pip to install scitools-iris (PyPI name) due to all the dependencies not being available on PyPI e.g., proj, udunits2 etc.

However, we now host pyke as scitools-pyke on PyPI (but the hope is to purge pyke as technical debt ASAP 🙏).

This PR introduces the pyproject.toml file to support PEP517 and PEP518 building during installation, thus allowing the pyke rules for iris to be built when using pip - this wasn't possible before.

Note that, all the requirements/*.txt files in conjunction with the setup.py file service installation of scitools-iris and it's extras, namely [all], [docs], and [test], pretty much as before.

[bjlittle]

Dropping the use of extensions as cartopy will now pull in gdal, and iris-grib is not used now with the conda requirements, so mirroring that here... hence, no extensions.

[bjlittle]

@tkknight This should be enough for readthedocs to find the associated yml file.

[pp-mo]

This is awesome + long overdue.
Right off the top of the head, a few general points arising for me...

• as the account here is so good, can we document it somehow ?
• we need to be clearer about the categories.
  • core/setup/test/docs are all good,
  • "extensions" was always peculiar.
  • But what is "all" for ??
• do we really have to have multiple files for different python versions??
  Yuk, frankly 🤢

[bjlittle]

@pp-mo Yeah, I've been meaning to do this for a long, long time... but just never got around to it, but glad I finally did 🎉

I agree with everything in your comment above 👍

This definitely should be documented and it will. I've left this for @tkknight to do as a follow up task, as he's totally swarming all over the documentation (mucho ❤️), and this PR unblocks him. I suspect that my description detail above may serve as a strawman to some developer notes in the follow-up PR.

Also, yeah, having the py36.yml and py37.yml is a slight pimple on the princess, but other projects do also take this approach e.g., see here.

To be honest, I'm willing to live with this and it does give us the scope to have dependencies per Python version, and we can maintain those dependencies historically going forwards, which wasn't possible before... so there are benefits, and at the end of the day I think I can see past the 🤢 if you can?

[tkknight]

Hey @bjlittle. Here is the PR (#3817) to bring the documentation inline with this PR. It will be in draft until this PR is merged.

[bjlittle]

@pp-mo Also, I'm keen that we use conda env create rather than conda create, as it's opinionated in terms of naming the environment and the channel that it pulls packages from, plus it's a very simple instruction to the user/developer, see #3817... with no scope for if'ing or but'ing about environment names, conda channels or Python versions.

[tkknight]

I'm happy to merge this now. Any last comments @pp-mo ?

[pp-mo]

@tkknight I'm happy to merge this now. Any last comments @pp-mo ?

No, after some thought I'm 👍