Merge branch 'main' of github.com:ESMValGroup/ESMValCore into dask-di…

…stributed
ESMValGroup · Nov 21, 2022 · 05d69f7 · 05d69f7
2 parents b455bbb + a6f8c08
commit 05d69f7
Show file tree

Hide file tree

Showing 100 changed files with 1,501 additions and 1,068 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -45,6 +45,10 @@ repos:
     hooks:
       - id: flake8
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: 'v0.812'
+    rev: 'v0.990'
     hooks:
       - id: mypy
+        additional_dependencies:
+          - 'types-PyYAML'
+          - 'types-pkg_resources'
+          - 'types-requests'
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -10,19 +10,15 @@ build:
   os: ubuntu-22.04
   tools:
     python: "mambaforge-4.10"
+  jobs:
+    post_create_environment:
+      - pip install . --no-deps
 
 # Declare the requirements required to build your docs
 conda:
   environment:
     environment.yml
 
-python:
-  install:
-    - method: pip
-      path: .
-      extra_requirements:
-        - doc
-
 # Build documentation in the doc directory with Sphinx
 sphinx:
   configuration: doc/conf.py

diff --git a/CITATION.cff b/CITATION.cff
@@ -188,11 +188,11 @@ authors:
     orcid: "https://orcid.org/0000-0002-5317-1247"
 
 cff-version: 1.2.0
-date-released: 2022-07-15
+date-released: 2022-10-13
 doi: "10.5281/zenodo.3387139"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValCore/"
 title: ESMValCore
-version: "v2.6.0"
+version: "v2.7.0"
 ...
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@
 [![codecov](https://codecov.io/gh/ESMValGroup/ESMValCore/branch/main/graph/badge.svg?token=wQnDzguwq6)](https://codecov.io/gh/ESMValGroup/ESMValCore)
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/5d496dea9ef64ec68e448a6df5a65783)](https://www.codacy.com/gh/ESMValGroup/ESMValCore?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=ESMValGroup/ESMValCore&amp;utm_campaign=Badge_Grade)
 [![Docker Build Status](https://img.shields.io/docker/cloud/build/esmvalgroup/esmvalcore)](https://hub.docker.com/r/esmvalgroup/esmvalcore/)
-[![Anaconda-Server Badge](https://anaconda.org/esmvalgroup/esmvalcore/badges/installer/conda.svg)](https://conda.anaconda.org/esmvalgroup)
+[![Anaconda-Server Badge](https://anaconda.org/conda-forge/esmvalcore/badges/version.svg)](https://anaconda.org/conda-forge/esmvalcore)
 [![Github Actions Test](https://github.com/ESMValGroup/ESMValCore/actions/workflows/run-tests.yml/badge.svg)](https://github.com/ESMValGroup/ESMValCore/actions/workflows/run-tests.yml)
 [![Github Actions Dashboard](https://api.meercode.io/badge/ESMValGroup/ESMValCore?type=ci-success-rate&branch=main&lastDay=14)](https://meercode.io)
 

diff --git a/conda-linux-64.lock b/conda-linux-64.lock
diff --git a/doc/api/esmvalcore.api.rst b/doc/api/esmvalcore.api.rst
diff --git a/doc/api/esmvalcore.api.config.rst → doc/api/esmvalcore.config.rst b/doc/api/esmvalcore.api.config.rst → doc/api/esmvalcore.config.rst
@@ -1,19 +1,17 @@
-.. _api_config:
-
 Configuration
 =============
 
-This section describes the :py:class:`~esmvalcore.experimental.config` submodule of the API (:py:mod:`esmvalcore.experimental`).
+This section describes the :py:class:`~esmvalcore.config` module.
 
 Config
 ******
 
-Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.experimental.config.Config` object.
-The global configuration can be imported from the :py:mod:`esmvalcore.experimental` module as :py:data:`~esmvalcore.experimental.CFG`:
+Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.config.Config` object.
+The global configuration can be imported from the :py:mod:`esmvalcore.config` module as :py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
-    >>> from esmvalcore.experimental import CFG
+    >>> from esmvalcore.config import CFG
     >>> CFG
     Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
             'compress_netcdf': False,
@@ -34,7 +32,7 @@ The global configuration can be imported from the :py:mod:`esmvalcore.experiment
 
 The parameters for the user configuration file are listed :ref:`here <user configuration file>`.
 
-:py:data:`~esmvalcore.experimental.CFG` is essentially a python dictionary with a few extra functions, similar to :py:mod:`matplotlib.rcParams`.
+:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few extra functions, similar to :py:data:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -43,7 +41,7 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that :py:data:`~esmvalcore.experimental.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
 All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
@@ -58,7 +56,7 @@ Or, if the value entered cannot be converted to the expected type:
     >>> CFG['max_parallel_tasks'] = '🐜'
     InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-:py:class:`~esmvalcore.experimental.config.Config` is also flexible, so it tries to correct the type of your input if possible:
+:py:class:`~esmvalcore.config.Config` is also flexible, so it tries to correct the type of your input if possible:
 
 .. code-block:: python
 
@@ -85,16 +83,16 @@ Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the :py:data:`~esmvalcore.experimental.config.Session` object.
-A :py:data:`~esmvalcore.experimental.config.Session` can be initiated from the global :py:class:`~esmvalcore.experimental.config.Config`.
+This behaviour can be controlled via the :py:data:`~esmvalcore.config.Session` object.
+A :py:data:`~esmvalcore.config.Session` can be initiated from the global :py:class:`~esmvalcore.config.Config`.
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
-A :py:data:`~esmvalcore.experimental.config.Session` is very similar to the config.
-It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.experimental.config.Config`.
-At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.experimental.CFG`:
+A :py:data:`~esmvalcore.config.Session` is very similar to the config.
+It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.config.Config`.
+At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
@@ -104,7 +102,7 @@ At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.exper
     >>> print(session == CFG)  # False
     False
 
-A :py:data:`~esmvalcore.experimental.config.Session` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.config.Session` also knows about the directories where the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -120,12 +118,12 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.experimental.config.Config`.
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.config.Config`.
 
 
 API reference
 *************
 
-.. automodule:: esmvalcore.experimental.config
+.. automodule:: esmvalcore.config
     :no-inherited-members:
     :no-show-inheritance:
diff --git a/doc/api/esmvalcore.experimental.config.rst b/doc/api/esmvalcore.experimental.config.rst
@@ -0,0 +1,6 @@
+.. _api_config:
+
+Configuration
+=============
+
+The :py:mod:`~esmvalcore.experimental.config` module has been moved to :py:mod:`esmvalcore.config`.
diff --git a/doc/api/esmvalcore.api.recipe.rst → doc/api/esmvalcore.experimental.recipe.rst b/doc/api/esmvalcore.api.recipe.rst → doc/api/esmvalcore.experimental.recipe.rst
diff --git a/doc/api/esmvalcore.api.recipe_metadata.rst → ...mvalcore.experimental.recipe_metadata.rst b/doc/api/esmvalcore.api.recipe_metadata.rst → ...mvalcore.experimental.recipe_metadata.rst
diff --git a/doc/api/esmvalcore.api.recipe_output.rst → ...esmvalcore.experimental.recipe_output.rst b/doc/api/esmvalcore.api.recipe_output.rst → ...esmvalcore.experimental.recipe_output.rst
diff --git a/doc/api/esmvalcore.experimental.rst b/doc/api/esmvalcore.experimental.rst
@@ -0,0 +1,17 @@
+.. _experimental_api:
+
+Experimental API
+================
+
+This page describes the new ESMValCore API.
+The experimental API module is available in the submodule ``esmvalcore.experimental``.
+The API is under development, so use at your own risk!
+
+.. toctree::
+   :maxdepth: 1
+
+   esmvalcore.experimental.config
+   esmvalcore.experimental.recipe
+   esmvalcore.experimental.recipe_output
+   esmvalcore.experimental.recipe_metadata
+   esmvalcore.experimental.utils
diff --git a/doc/api/esmvalcore.api.utils.rst → doc/api/esmvalcore.experimental.utils.rst b/doc/api/esmvalcore.api.utils.rst → doc/api/esmvalcore.experimental.utils.rst
diff --git a/doc/api/esmvalcore.rst b/doc/api/esmvalcore.rst
@@ -7,10 +7,12 @@ ESMValCore is mostly used as a commandline tool. However, it is also possibly to
 library. This section documents the public API of ESMValCore.
 
 .. toctree::
+   :maxdepth: 1
 
    esmvalcore.cmor
+   esmvalcore.config
    esmvalcore.esgf
    esmvalcore.exceptions
    esmvalcore.iris_helpers
    esmvalcore.preprocessor
-   esmvalcore.api
+   esmvalcore.experimental
diff --git a/doc/changelog.rst b/doc/changelog.rst
@@ -29,6 +29,7 @@ Bug fixes
 -  Core `_io.concatenate()` may fail due to case when one of the cubes is scalar - this fixes that (`#1715 <https://github.com/ESMValGroup/ESMValCore/pull/1715>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
 -  Pick up esmvalcore badge instead of esmvaltool one in README (`#1749 <https://github.com/ESMValGroup/ESMValCore/pull/1749>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
 -  Restore support for scalar cubes to time selection preprocessor functions (`#1750 <https://github.com/ESMValGroup/ESMValCore/pull/1750>`__) `Bouwe Andela <https://github.com/bouweandela>`__
+-  Fix calculation of precipitation flux in EMAC on-the-fly CMORizer (`#1755 <https://github.com/ESMValGroup/ESMValCore/pull/1755>`__) `Manuel Schlund <https://github.com/schlunma>`__
 
 Deprecations
 ~~~~~~~~~~~~
@@ -42,6 +43,8 @@ Documentation
 -  Add MO-paths to config file (`#1709 <https://github.com/ESMValGroup/ESMValCore/pull/1709>`__) `mo-tgeddes <https://github.com/mo-tgeddes>`__
 -  Update MO obs4MIPs paths in the user configuration file (`#1734 <https://github.com/ESMValGroup/ESMValCore/pull/1734>`__) `mo-tgeddes <https://github.com/mo-tgeddes>`__
 -  Update `Making a release` section of the documentation (`#1689 <https://github.com/ESMValGroup/ESMValCore/pull/1689>`__) `sloosvel <https://github.com/sloosvel>`__
+-  Added changelog for v2.7.0 (`#1746 <https://github.com/ESMValGroup/ESMValCore/pull/1746>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
+-  update CITATION.cff file with 2.7.0 release info (`#1757 <https://github.com/ESMValGroup/ESMValCore/pull/1757>`__) `Valeriu Predoi <https://github.com/valeriupredoi>`__
 
 Improvements
 ~~~~~~~~~~~~

diff --git a/doc/conf.py b/doc/conf.py
@@ -33,7 +33,7 @@
 
 # This is used for linking and such so we link to the thing we're building
 rtd_version = os.environ.get("READTHEDOCS_VERSION", "latest")
-if rtd_version not in ["latest", "doc"]:  # TODO: add "stable" once we have it
+if rtd_version not in ["latest", "stable", "doc"]:
     rtd_version = "latest"
 
 # -- General configuration ------------------------------------------------

diff --git a/doc/contributing.rst b/doc/contributing.rst
@@ -635,13 +635,13 @@ When adding or removing dependencies, please consider applying the changes in
 the following files:
 
 - ``environment.yml``
-  contains development dependencies that cannot be installed from
-  `PyPI <https://pypi.org/>`_
+  contains all the development dependencies; these are all from
+  `conda-forge <https://conda-forge.org/>`_
 - ``setup.py``
   contains all Python dependencies, regardless of their installation source
 
 Note that packages may have a different name on
-`conda-forge <https://conda-forge.org/>`__ than on PyPI_.
+`conda-forge <https://conda-forge.org/>`__ than on `PyPI <https://pypi.org/>`_.
 
 Several test jobs on CircleCI_ related to the installation of the tool will only
 run if you change the dependencies.

diff --git a/doc/develop/fixing_data.rst b/doc/develop/fixing_data.rst
@@ -441,7 +441,7 @@ To allow ESMValCore to locate the data files, use the following steps:
      Note that it is possible to predefine facets in an :ref:`extra facets file
      <add_new_fix_native_datasets_extra_facets>`.
      In this ICON example, the facet ``var_type`` is :download:`predefined
-     </../esmvalcore/_config/extra_facets/icon-mappings.yml>` for many
+     </../esmvalcore/config/extra_facets/icon-mappings.yml>` for many
      variables.
 
 .. _add_new_fix_native_datasets_fix_data:
@@ -472,7 +472,7 @@ e.g. with variable naming issues for finding files or additional information
 that is required for the fixes.
 See :ref:`extra_facets` and :ref:`extra-facets-fixes` for more details on this.
 An example of such a file for IPSL-CM6 is given :download:`here
-<../../esmvalcore/_config/extra_facets/ipslcm-mappings.yml>`.
+<../../esmvalcore/config/extra_facets/ipslcm-mappings.yml>`.
 
 
 .. _extra-facets-fixes:

diff --git a/doc/quickstart/configure.rst b/doc/quickstart/configure.rst
@@ -754,7 +754,7 @@ For example, this is used to automatically add ``product: output1`` to any
 variable of any CMIP5 dataset that does not have a ``product`` key yet:
 
 .. code-block:: yaml
-   :caption: Extra facet example file `cmip5-product.yml <https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/_config/extra_facets/cmip5-product.yml>`_
+   :caption: Extra facet example file `cmip5-product.yml <https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/config/extra_facets/cmip5-product.yml>`_
 
    '*':
      '*':
@@ -765,15 +765,15 @@ Location of the extra facets files
 Extra facets files can be placed in several different places. When we use them
 to support a particular use-case within the ESMValTool project, they will be
 provided in the sub-folder `extra_facets` inside the package
-`esmvalcore._config`. If they are used from the user side, they can be either
+:mod:`esmvalcore.config`. If they are used from the user side, they can be either
 placed in `~/.esmvaltool/extra_facets` or in any other directory of the users
 choosing. In that case this directory must be added to the `config-user.yml`
 file under the `extra_facets_dir` setting, which can take a single directory or
 a list of directories.
 
 The order in which the directories are searched is
 
-1. The internal directory `esmvalcore._config/extra_facets`
+1. The internal directory `esmvalcore.config/extra_facets`
 2. The default user directory `~/.esmvaltool/extra_facets`
 3. The custom user directories in the order in which they are given in
    `config-user.yml`.

diff --git a/doc/quickstart/find_data.rst b/doc/quickstart/find_data.rst
@@ -202,7 +202,7 @@ overwritten in the recipe.
 Similar to any other fix, the CESM fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`cesm-mappings.yml
-</../esmvalcore/_config/extra_facets/cesm-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/cesm-mappings.yml>` is used for that
 purpose.
 Currently, this file only contains default facets for a single variable
 (`tas`); for other variables, these entries need to be defined in the recipe.
@@ -269,7 +269,7 @@ the recipe.
 Similar to any other fix, the EMAC fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`emac-mappings.yml
-</../esmvalcore/_config/extra_facets/emac-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/emac-mappings.yml>` is used for that
 purpose.
 For some variables, extra facets are necessary; otherwise ESMValTool cannot
 read them properly.
@@ -340,7 +340,7 @@ the recipe.
 Similar to any other fix, the ICON fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`icon-mappings.yml
-</../esmvalcore/_config/extra_facets/icon-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/icon-mappings.yml>` is used for that
 purpose.
 For some variables, extra facets are necessary; otherwise ESMValTool cannot
 read them properly.
@@ -405,7 +405,7 @@ The ``Output`` format is an example of a case where variables are grouped in
 multi-variable files, which name cannot be computed directly from datasets
 attributes alone but requires to use an extra_facets file, which principles are
 explained in :ref:`extra_facets`, and which content is :download:`available here
-</../esmvalcore/_config/extra_facets/ipslcm-mappings.yml>`. These multi-variable
+</../esmvalcore/config/extra_facets/ipslcm-mappings.yml>`. These multi-variable
 files must also undergo some data selection.