Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed search for fx files when no mip is given #1216

Merged
merged 9 commits into from
Jul 12, 2021
Merged

Fixed search for fx files when no mip is given #1216

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
368b017
Fixed tests related to fx file retrieval
schlunma Jul 6, 2021
4cca1e5
Raise error when fx variable is present in multiple tables
schlunma Jul 7, 2021
a9b85da
Update doc/recipe/preprocessor.rst
schlunma Jul 7, 2021
2e0d9dc
Updated documentation on fx variables
schlunma Jul 8, 2021
cc60332
Used :ref: to link area_statistics and volume_statistics
schlunma Jul 8, 2021
9dfd16d
Optimized fx doc
schlunma Jul 8, 2021
ca039cf
Only raise ambiguity error if fx files are found in mutiple tables
schlunma Jul 8, 2021
e49ddb5
Merge remote-tracking branch 'origin/main' into fix_fx_test
schlunma Jul 8, 2021
8cd5dcf
Fixed machine-dependent error-message
schlunma Jul 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 109 additions & 102 deletions doc/recipe/preprocessor.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,22 +180,85 @@ To get an overview on data fixes and how to implement new ones, please go to

zklaus marked this conversation as resolved.
Show resolved Hide resolved
Fx variables as cell measures or ancillary variables
====================================================
The following preprocessor may require the use of ``fx_variables``
to be able to perform the computations:
The following preprocessors may require the use of ``fx_variables`` to be able
to perform the computations:

============================================================== =====================
Preprocessor Default fx variables
============================================================== =====================
:ref:`area_statistics<area_statistics>` ``areacella``, ``areacello``
:ref:`mask_landsea<land/sea/ice masking>` ``sftlf``, ``sftof``
:ref:`mask_landseaice<ice masking>` ``sftgif``
:ref:`volume_statistics<volume_statistics>` ``volcello``
:ref:`weighting_landsea_fraction<land/sea fraction weighting>` ``sftlf``, ``sftof``
============================================================== =====================

If no ``fx_variables`` are specified for these preprocessors, the fx variables
in the second column are used. If given, the ``fx_variables`` argument
specifies the fx variables that the user wishes to input to the corresponding
preprocessor function. The user may specify these by simply adding the names of
the variables, e.g.,

- ``area_statistics``
- ``mask_landsea``
- ``mask_landseaice``
- ``volume_statistics``
- ``weighting_landsea_fraction``
.. code-block:: yaml

fx_variables:
areacello:
volcello:

or by additionally specifying further keys that are used to define the fx
datasets, e.g.,

.. code-block:: yaml

fx_variables:
areacello:
mip: Ofx
exp: piControl
volcello:
mip: Omon

This might be useful to select fx files from a specific ``mip`` table or from a
specific ``exp`` in case not all experiments provide the fx variable.

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

The preprocessor step ``add_fx_variables`` loads the required ``fx_variables``,
checks them against CMOR standards and adds them either as ``cell_measure``
or ``ancillary_variable`` inside the cube data. This ensures that the
defined preprocessor chain is applied to both ``variables`` and ``fx_variables``.
fx_variables: [{'short_name': 'areacello', 'mip': 'Ofx', 'exp': 'piControl'}, {'short_name': 'volcello', 'mip': 'Omon'}]

The recipe parser will automatically find the data files that are associated
with these variables and pass them to the function for loading and processing.

If ``mip`` is not given, ESMValTool will search for the fx variable in all
available tables of the specified project.

.. warning::
Some fx variables exist in more than one table (e.g., ``volcello`` exists in
CMIP6's ``Odec``, ``Ofx``, ``Omon``, and ``Oyr`` tables; ``sftgif`` exists
in CMIP6's ``fx``, ``IyrAnt`` and ``IyrGre``, and ``LImon`` tables). If (for
a given dataset) fx files are found in more than one table, ``mip`` needs to
be specified, otherwise an error is raised.

Internally, the required ``fx_variables`` are automatically loaded by the
preprocessor step ``add_fx_variables`` which also checks them against CMOR
standards and adds them either as ``cell_measure`` (see `CF conventions on cell
measures
<https://cfconventions.org/cf-conventions/cf-conventions.html#cell-measures>`_
and :class:`iris.coords.CellMeasure`) or ``ancillary_variable`` (see `CF
conventions on ancillary variables
<https://cfconventions.org/cf-conventions/cf-conventions.html#ancillary-data>`_
and :class:`iris.coords.AncillaryVariable`) inside the cube data. This ensures
that the defined preprocessor chain is applied to both ``variables`` and
``fx_variables``.

Note that when calling steps that require ``fx_variables`` inside diagnostic
scripts, the variables are expected to contain the required ``cell_measures`` or
scripts, the variables are expected to contain the required ``cell_measures`` or
``ancillary_variables``. If missing, they can be added using the following functions:

.. code-block::
Expand All @@ -209,7 +272,7 @@ scripts, the variables are expected to contain the required ``cell_measures`` or
cube_with_ancillary_sftlf = add_ancillary_variable(cube, sftlf_cube)

cube_with_ancillary_sftgif = add_ancillary_variable(cube, sftgif_cube)

Details on the arguments needed for each step can be found in the following sections.

.. _Vertical interpolation:
Expand Down Expand Up @@ -351,8 +414,8 @@ is for example useful for climate models which do not offer land/sea fraction
files. This arguments also accepts the special dataset specifiers
``reference_dataset`` and ``alternative_dataset``.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval:
Optionally you can specify your own custom fx variable to be used in cases when
e.g. a certain experiment is preferred for fx data retrieval:

.. code-block:: yaml

Expand All @@ -361,7 +424,7 @@ experiment is preferred for fx data retrieval:
weighting_landsea_fraction:
area_type: land
exclude: ['CanESM2', 'reference_dataset']
fx_variables:
fx_variables:
sftlf:
exp: piControl
sftof:
Expand All @@ -377,10 +440,13 @@ or alternatively:
area_type: land
exclude: ['CanESM2', 'reference_dataset']
fx_variables: [
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftof', 'exp': 'piControl'}
]

More details on the argument ``fx_variables`` and its default values are given
in :ref:`Fx variables as cell measures or ancillary variables`.

See also :func:`esmvalcore.preprocessor.weighting_landsea_fraction`.


Expand Down Expand Up @@ -426,11 +492,6 @@ To mask out a certain domain (e.g., sea) in the preprocessor,

and requires only one argument: ``mask_out``: either ``land`` or ``sea``.

The preprocessor automatically retrieves the corresponding mask (``fx: stfof``
in this case) and applies it so that sea-covered grid cells are set to
missing. Conversely, it retrieves the ``fx: sftlf`` mask when land needs to be
masked out, respectively.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval. Note that it is possible to specify as many tags
for the fx variable as required:
Expand All @@ -442,8 +503,8 @@ for the fx variable as required:
landmask:
mask_landsea:
mask_out: sea
fx_variables:
sftlf:
fx_variables:
sftlf:
exp: piControl
sftof:
exp: piControl
Expand All @@ -458,10 +519,13 @@ or alternatively:
mask_landsea:
mask_out: sea
fx_variables: [
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftof', 'exp': 'piControl', 'ensemble': 'r2i1p1f1'}
]

More details on the argument ``fx_variables`` and its default values are given
in :ref:`Fx variables as cell measures or ancillary variables`.

If the corresponding fx file is not found (which is
the case for some models and almost all observational datasets), the
preprocessor attempts to mask the data using Natural Earth mask files (that are
Expand All @@ -471,6 +535,8 @@ land and glaciated areas and 50m for ocean masks).

See also :func:`esmvalcore.preprocessor.mask_landsea`.

.. _ice masking:

Ice masking
-----------

Expand All @@ -487,11 +553,8 @@ losing generality. To mask ice out, ``mask_landseaice`` can be used:

and requires only one argument: ``mask_out``: either ``landsea`` or ``ice``.

As in the case of ``mask_landsea``, the preprocessor automatically retrieves
the ``fx_variables: [sftgif]`` mask.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval:
Optionally you can specify your own custom fx variable to be used in cases when
e.g. a certain experiment is preferred for fx data retrieval:


.. code-block:: yaml
Expand All @@ -500,7 +563,7 @@ experiment is preferred for fx data retrieval:
landseaicemask:
mask_landseaice:
mask_out: sea
fx_variables:
fx_variables:
sftgif:
exp: piControl

Expand All @@ -514,6 +577,9 @@ or alternatively:
mask_out: sea
fx_variables: [{'short_name': 'sftgif', 'exp': 'piControl'}]

More details on the argument ``fx_variables`` and its default values are given
in :ref:`Fx variables as cell measures or ancillary variables`.

See also :func:`esmvalcore.preprocessor.mask_landseaice`.

Glaciated masking
Expand Down Expand Up @@ -1021,7 +1087,7 @@ See also :func:`esmvalcore.preprocessor.decadal_statistics`.
----------------------

This function produces statistics for the whole dataset. It can produce scalars
(if the full period is chosen) or daily, monthly or seasonal statics.
(if the full period is chosen) or daily, monthly or seasonal statistics.

Parameters:
* operator: operation to apply. Accepted values are 'mean', 'median',
Expand Down Expand Up @@ -1406,6 +1472,8 @@ argument:
See also :func:`esmvalcore.preprocessor.meridional_means`.


.. _area_statistics:

``area_statistics``
-------------------

Expand All @@ -1420,40 +1488,9 @@ Note that this function is applied over the entire dataset. If only a specific
region, depth layer or time period is required, then those regions need to be
removed using other preprocessor operations in advance.

The ``fx_variables`` argument specifies the fx variables that the user wishes to input to the function;
the user may specify it calling the variables e.g.

.. code-block:: yaml

fx_variables:
areacello:
volcello:

or calling the variables and adding specific variable parameters (the key-value pair may be as specific
as a CMOR variable can permit):

.. code-block:: yaml

fx_variables:
areacello:
mip: Omon
volcello:
mip: fx

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

fx_variables: [{'short_name': 'areacello', 'mip': 'Omon'}, {'short_name': 'volcello', 'mip': 'fx'}]

The recipe parser will automatically find the data files that are associated with these
variables and pass them to the function for loading and processing.
The optional ``fx_variables`` argument specifies the fx variables that the user
wishes to input to the function. More details on this are given in :ref:`Fx
variables as cell measures or ancillary variables`.

See also :func:`esmvalcore.preprocessor.area_statistics`.

Expand Down Expand Up @@ -1486,6 +1523,8 @@ as the Iris cube. That is, if the cube has `z`-coordinate as negative, then
See also :func:`esmvalcore.preprocessor.extract_volume`.


.. _volume_statistics:

``volume_statistics``
---------------------

Expand All @@ -1496,42 +1535,10 @@ This function takes the argument: ``operator``, which defines the operation to
apply over the volume.

No depth coordinate is required as this is determined by Iris. This function
works best when the ``fx_variables`` provide the cell volume.

The ``fx_variables`` argument specifies the fx variables that the user wishes to input to the function;
the user may specify it calling the variables e.g.

.. code-block:: yaml

fx_variables:
areacello:
volcello:

or calling the variables and adding specific variable parameters (the key-value pair may be as specific
as a CMOR variable can permit):

.. code-block:: yaml

fx_variables:
areacello:
mip: Omon
volcello:
mip: fx

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

fx_variables: [{'short_name': 'areacello', 'mip': 'Omon'}, {'short_name': 'volcello', 'mip': 'fx'}]

The recipe parser will automatically find the data files that are associated with these
variables and pass them to the function for loading and processing.
works best when the ``fx_variables`` provide the cell volume. The optional
``fx_variables`` argument specifies the fx variables that the user wishes to
input to the function. More details on this are given in :ref:`Fx variables as
cell measures or ancillary variables`.

See also :func:`esmvalcore.preprocessor.volume_statistics`.

Expand Down
Loading