Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed search for fx files when no mip is given #1216

Merged
merged 9 commits into from
Jul 12, 2021
Merged

Fixed search for fx files when no mip is given #1216

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
368b017
Fixed tests related to fx file retrieval
schlunma Jul 6, 2021
4cca1e5
Raise error when fx variable is present in multiple tables
schlunma Jul 7, 2021
a9b85da
Update doc/recipe/preprocessor.rst
schlunma Jul 7, 2021
2e0d9dc
Updated documentation on fx variables
schlunma Jul 8, 2021
cc60332
Used :ref: to link area_statistics and volume_statistics
schlunma Jul 8, 2021
9dfd16d
Optimized fx doc
schlunma Jul 8, 2021
ca039cf
Only raise ambiguity error if fx files are found in mutiple tables
schlunma Jul 8, 2021
e49ddb5
Merge remote-tracking branch 'origin/main' into fix_fx_test
schlunma Jul 8, 2021
8cd5dcf
Fixed machine-dependent error-message
schlunma Jul 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 84 additions & 98 deletions doc/recipe/preprocessor.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,22 +180,68 @@ To get an overview on data fixes and how to implement new ones, please go to

zklaus marked this conversation as resolved.
Show resolved Hide resolved
Fx variables as cell measures or ancillary variables
====================================================
The following preprocessor may require the use of ``fx_variables``
to be able to perform the computations:
The following preprocessor may require the use of ``fx_variables`` to be able
schlunma marked this conversation as resolved.
Show resolved Hide resolved
to perform the computations:

- ``area_statistics``
- ``mask_landsea``
- ``mask_landseaice``
- ``volume_statistics``
- ``weighting_landsea_fraction``
- area_statistics_ (``areacella``, ``areacello``)
- :ref:`mask_landsea<land/sea/ice masking>` (``sftlf``, ``sftof``)
- :ref:`mask_landseaice<ice masking>` (``sftgif``)
- volume_statistics_ (``volcello``)
- :ref:`weighting_landsea_fraction<land/sea fraction weighting>` (``sftlf``, ``sftof``)
schlunma marked this conversation as resolved.
Show resolved Hide resolved

If no ``fx_variables`` are specified for these preprocessors, the fx variables
given in brackets are used. If given, the ``fx_variables`` argument specifies
the fx variables that the user wishes to input to the corresponding
preprocessor function; the user may specify it calling the variables e.g.

.. code-block:: yaml

fx_variables:
areacello:
volcello:

or calling the variables and adding specific variable parameters (the key-value
pair may be as specific as a CMOR variable can permit):
schlunma marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: yaml

fx_variables:
areacello:
mip: Omon
volcello:
mip: fx
schlunma marked this conversation as resolved.
Show resolved Hide resolved

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

fx_variables: [{'short_name': 'areacello', 'mip': 'Omon'}, {'short_name': 'volcello', 'mip': 'fx'}]
schlunma marked this conversation as resolved.
Show resolved Hide resolved

The recipe parser will automatically find the data files that are associated
with these variables and pass them to the function for loading and processing.

If ``mip`` is not given, ESMValTool will search for the fx variable in all
available tables of the specified project.

.. note::
schlunma marked this conversation as resolved.
Show resolved Hide resolved
Some fx variables exist in more than one table (e.g., ``volcello`` exists in
CMIP6's ``Ofx``, ``Oyr``, ``Odec`` and ``Omon`` tables or ``sftgif`` exists
schlunma marked this conversation as resolved.
Show resolved Hide resolved
in CMIP6's ``fx``, ``LImon``, ``IyrGre`` and ``IyrAnt`` tables). In these
schlunma marked this conversation as resolved.
Show resolved Hide resolved
cases, ``mip`` needs to be specified, otherwise an error is raised.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

The preprocessor step ``add_fx_variables`` loads the required ``fx_variables``,
schlunma marked this conversation as resolved.
Show resolved Hide resolved
checks them against CMOR standards and adds them either as ``cell_measure``
schlunma marked this conversation as resolved.
Show resolved Hide resolved
or ``ancillary_variable`` inside the cube data. This ensures that the
schlunma marked this conversation as resolved.
Show resolved Hide resolved
defined preprocessor chain is applied to both ``variables`` and ``fx_variables``.

Note that when calling steps that require ``fx_variables`` inside diagnostic
scripts, the variables are expected to contain the required ``cell_measures`` or
scripts, the variables are expected to contain the required ``cell_measures`` or
``ancillary_variables``. If missing, they can be added using the following functions:

.. code-block::
Expand All @@ -209,7 +255,7 @@ scripts, the variables are expected to contain the required ``cell_measures`` or
cube_with_ancillary_sftlf = add_ancillary_variable(cube, sftlf_cube)

cube_with_ancillary_sftgif = add_ancillary_variable(cube, sftgif_cube)

Details on the arguments needed for each step can be found in the following sections.

.. _Vertical interpolation:
Expand Down Expand Up @@ -351,8 +397,8 @@ is for example useful for climate models which do not offer land/sea fraction
files. This arguments also accepts the special dataset specifiers
``reference_dataset`` and ``alternative_dataset``.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval:
Optionally you can specify your own custom fx variable to be used in cases when
e.g. a certain experiment is preferred for fx data retrieval:

.. code-block:: yaml

Expand All @@ -361,7 +407,7 @@ experiment is preferred for fx data retrieval:
weighting_landsea_fraction:
area_type: land
exclude: ['CanESM2', 'reference_dataset']
fx_variables:
fx_variables:
sftlf:
exp: piControl
sftof:
Expand All @@ -377,10 +423,13 @@ or alternatively:
area_type: land
exclude: ['CanESM2', 'reference_dataset']
fx_variables: [
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftof', 'exp': 'piControl'}
]

More details on the argument ``fx_variables`` and its default values are given
:ref:`here<Fx variables as cell measures or ancillary variables>`.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

See also :func:`esmvalcore.preprocessor.weighting_landsea_fraction`.


Expand Down Expand Up @@ -426,11 +475,6 @@ To mask out a certain domain (e.g., sea) in the preprocessor,

and requires only one argument: ``mask_out``: either ``land`` or ``sea``.

The preprocessor automatically retrieves the corresponding mask (``fx: stfof``
in this case) and applies it so that sea-covered grid cells are set to
missing. Conversely, it retrieves the ``fx: sftlf`` mask when land needs to be
masked out, respectively.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval. Note that it is possible to specify as many tags
for the fx variable as required:
Expand All @@ -442,8 +486,8 @@ for the fx variable as required:
landmask:
mask_landsea:
mask_out: sea
fx_variables:
sftlf:
fx_variables:
sftlf:
exp: piControl
sftof:
exp: piControl
Expand All @@ -458,10 +502,13 @@ or alternatively:
mask_landsea:
mask_out: sea
fx_variables: [
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftlf', 'exp': 'piControl'},
{'short_name': 'sftof', 'exp': 'piControl', 'ensemble': 'r2i1p1f1'}
]

More details on the argument ``fx_variables`` and its default values are given
:ref:`here<Fx variables as cell measures or ancillary variables>`.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

If the corresponding fx file is not found (which is
the case for some models and almost all observational datasets), the
preprocessor attempts to mask the data using Natural Earth mask files (that are
Expand All @@ -471,6 +518,8 @@ land and glaciated areas and 50m for ocean masks).

See also :func:`esmvalcore.preprocessor.mask_landsea`.

.. _ice masking:

Ice masking
-----------

Expand All @@ -487,11 +536,8 @@ losing generality. To mask ice out, ``mask_landseaice`` can be used:

and requires only one argument: ``mask_out``: either ``landsea`` or ``ice``.

As in the case of ``mask_landsea``, the preprocessor automatically retrieves
the ``fx_variables: [sftgif]`` mask.

Optionally you can specify your own custom fx variable to be used in cases when e.g. a certain
experiment is preferred for fx data retrieval:
Optionally you can specify your own custom fx variable to be used in cases when
e.g. a certain experiment is preferred for fx data retrieval:


.. code-block:: yaml
Expand All @@ -500,7 +546,7 @@ experiment is preferred for fx data retrieval:
landseaicemask:
mask_landseaice:
mask_out: sea
fx_variables:
fx_variables:
sftgif:
exp: piControl

Expand All @@ -514,6 +560,9 @@ or alternatively:
mask_out: sea
fx_variables: [{'short_name': 'sftgif', 'exp': 'piControl'}]

More details on the argument ``fx_variables`` and its default values are given
:ref:`here<Fx variables as cell measures or ancillary variables>`.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

See also :func:`esmvalcore.preprocessor.mask_landseaice`.

Glaciated masking
Expand Down Expand Up @@ -1021,7 +1070,7 @@ See also :func:`esmvalcore.preprocessor.decadal_statistics`.
----------------------

This function produces statistics for the whole dataset. It can produce scalars
(if the full period is chosen) or daily, monthly or seasonal statics.
(if the full period is chosen) or daily, monthly or seasonal statistics.

Parameters:
* operator: operation to apply. Accepted values are 'mean', 'median',
Expand Down Expand Up @@ -1420,40 +1469,9 @@ Note that this function is applied over the entire dataset. If only a specific
region, depth layer or time period is required, then those regions need to be
removed using other preprocessor operations in advance.

The ``fx_variables`` argument specifies the fx variables that the user wishes to input to the function;
the user may specify it calling the variables e.g.

.. code-block:: yaml

fx_variables:
areacello:
volcello:

or calling the variables and adding specific variable parameters (the key-value pair may be as specific
as a CMOR variable can permit):

.. code-block:: yaml

fx_variables:
areacello:
mip: Omon
volcello:
mip: fx

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

fx_variables: [{'short_name': 'areacello', 'mip': 'Omon'}, {'short_name': 'volcello', 'mip': 'fx'}]

The recipe parser will automatically find the data files that are associated with these
variables and pass them to the function for loading and processing.
The optional ``fx_variables`` argument specifies the fx variables that the user
wishes to input to the function. More details on this are given
:ref:`here<Fx variables as cell measures or ancillary variables>`.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

See also :func:`esmvalcore.preprocessor.area_statistics`.

Expand Down Expand Up @@ -1496,42 +1514,10 @@ This function takes the argument: ``operator``, which defines the operation to
apply over the volume.

No depth coordinate is required as this is determined by Iris. This function
works best when the ``fx_variables`` provide the cell volume.

The ``fx_variables`` argument specifies the fx variables that the user wishes to input to the function;
the user may specify it calling the variables e.g.

.. code-block:: yaml

fx_variables:
areacello:
volcello:

or calling the variables and adding specific variable parameters (the key-value pair may be as specific
as a CMOR variable can permit):

.. code-block:: yaml

fx_variables:
areacello:
mip: Omon
volcello:
mip: fx

Alternatively, the ``fx_variables`` argument can also be specified as a list:

.. code-block:: yaml

fx_variables: ['areacello', 'volcello']

or as a list of dictionaries:

.. code-block:: yaml

fx_variables: [{'short_name': 'areacello', 'mip': 'Omon'}, {'short_name': 'volcello', 'mip': 'fx'}]

The recipe parser will automatically find the data files that are associated with these
variables and pass them to the function for loading and processing.
works best when the ``fx_variables`` provide the cell volume. The optional
``fx_variables`` argument specifies the fx variables that the user wishes to
input to the function. More details on this are given
:ref:`here<Fx variables as cell measures or ancillary variables>`.
schlunma marked this conversation as resolved.
Show resolved Hide resolved

See also :func:`esmvalcore.preprocessor.volume_statistics`.

Expand Down
Loading