Skip to content

Commit

Permalink
Address pull request comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@mats-knmi
mats-knmi committed Aug 12, 2024
1 parent 1fa4999 commit 369ef4a
Show file tree
Hide file tree
Showing 5 changed files with 137 additions and 34 deletions.
87 changes: 87 additions & 0 deletions pysteps/io/importers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,93 @@
| zr_b | the Z-R exponent b in Z = a*R**b |
+------------------+----------------------------------------------------------+
The data and metadata is then postprocessed into an xarray dataset. This dataset will
always contain an x and y dimension, but can be extended with a time dimension and/or
an ensemble member dimension over the course of the process.
The dataset can contain the following coordinate variables:
.. tabularcolumns:: |p{2cm}|L|
+---------------+-------------------------------------------------------------------------------------------+
| Coordinate | Description |
+===============+===========================================================================================+
| y | y-coordinate in Cartesian system, with units determined by ``metadata["cartesian_unit"]`` |
+---------------+-------------------------------------------------------------------------------------------+
| x | x-coordinate in Cartesian system, with units determined by ``metadata["cartesian_unit"]`` |
+---------------+-------------------------------------------------------------------------------------------+
| lat | latitude coordinate in degrees |
+---------------+-------------------------------------------------------------------------------------------+
| lon | longitude coordinate in degrees |
+---------------+-------------------------------------------------------------------------------------------+
| time | forecast time in seconds since forecast start time |
+---------------+-------------------------------------------------------------------------------------------+
| member | ensemble member number (integer) |
+---------------+-------------------------------------------------------------------------------------------+
The dataset can contain the following data variables:
.. tabularcolumns:: |p{2cm}|L|
+-------------------+-----------------------------------------------------------------------------------------------------------+
| Variable | Description |
+===================+===========================================================================================================+
| precip_intensity, | precipitation data, based on the unit the data has it is stored in one of these 3 possible variables |
| precip_accum | precip_intensity if unit is ``mm/h``, precip_accum if unit is ``mm`` and reflectivity if unit is ``dBZ``, |
| or reflectivity | the attributes of this variable contain metadata relevant to this attribute (see below) |
+-------------------+-----------------------------------------------------------------------------------------------------------+
| quality | value between 0 and 1 denoting the quality of the precipitation data, currently not used for anything |
+-------------------+-----------------------------------------------------------------------------------------------------------+
Some of the metadata in the metadata dictionary is not explicitely stored in the dataset,
but is still implicitly present. For example ``x1`` can easily be found by taking the first
value from the x coordinate variable. Metadata that is not implicitly present is explicitly
stored either in the datasets global attributes or as attributes of the precipitation variable.
Data that relates to the entire dataset is stored in the global attributes. The following data
is stored in the global attributes:
.. tabularcolumns:: |p{2cm}|L|
+------------------+----------------------------------------------------------+
| Key | Value |
+==================+==========================================================+
| projection | PROJ.4-compatible projection definition |
+------------------+----------------------------------------------------------+
| institution | name of the institution who provides the data |
+------------------+----------------------------------------------------------+
| precip_var | the name of the precipitation variable in this dataset |
+------------------+----------------------------------------------------------+
The following data is stored as attributes of the precipitation variable:
.. tabularcolumns:: |p{2cm}|L|
+------------------+----------------------------------------------------------+
| Key | Value |
+==================+==========================================================+
| units | the physical unit of the data: 'mm/h', 'mm' or 'dBZ' |
+------------------+----------------------------------------------------------+
| transform | the transformation of the data: None, 'dB', 'Box-Cox' or |
| | others |
+------------------+----------------------------------------------------------+
| accutime | the accumulation time in minutes of the data, float |
+------------------+----------------------------------------------------------+
| threshold | the rain/no rain threshold with the same unit, |
| | transformation and accutime of the data. |
+------------------+----------------------------------------------------------+
| zerovalue | the value assigned to the no rain pixels with the same |
| | unit, transformation and accutime of the data. |
+------------------+----------------------------------------------------------+
| zr_a | the Z-R constant a in Z = a*R**b |
+------------------+----------------------------------------------------------+
| zr_b | the Z-R exponent b in Z = a*R**b |
+------------------+----------------------------------------------------------+
Furthermore the dataset can contain some additional metadata to make the dataset
CF-compliant.
Available Importers
-------------------
Expand Down
2 changes: 1 addition & 1 deletion pysteps/tests/helpers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ def get_precipitation_fields(
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
"""

if source == "bom":
Expand Down
12 changes: 6 additions & 6 deletions pysteps/utils/conversion.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ def to_rainrate(dataset: xr.Dataset, zr_a=None, zr_b=None):
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be (back-)transformed.
Additionally, in case of conversion to/from reflectivity units, the
Expand All @@ -83,7 +83,7 @@ def to_rainrate(dataset: xr.Dataset, zr_a=None, zr_b=None):
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the converted units.
"""

Expand Down Expand Up @@ -159,7 +159,7 @@ def to_raindepth(dataset: xr.Dataset, zr_a=None, zr_b=None):
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be (back-)transformed.
Additionally, in case of conversion to/from reflectivity units, the
Expand All @@ -172,7 +172,7 @@ def to_raindepth(dataset: xr.Dataset, zr_a=None, zr_b=None):
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the converted units.
"""

Expand Down Expand Up @@ -248,7 +248,7 @@ def to_reflectivity(dataset: xr.Dataset, zr_a=None, zr_b=None):
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be (back-)transformed.
Additionally, in case of conversion to/from reflectivity units, the
Expand All @@ -261,7 +261,7 @@ def to_reflectivity(dataset: xr.Dataset, zr_a=None, zr_b=None):
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the converted units.
"""

Expand Down
54 changes: 35 additions & 19 deletions pysteps/utils/dimension.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,21 @@
)


def aggregate_fields_time(dataset: xr.Dataset, time_window_min, ignore_nan=False):
def aggregate_fields_time(
dataset: xr.Dataset, time_window_min, ignore_nan=False
) -> xr.Dataset:
"""Aggregate fields in time.
It attempts to aggregate the given dataset in the time direction in an integer
number of sections of length = ``time_window_min``.
If such a aggregation is not possible, an error is raised.
The data is aggregated by a method chosen based on the unit of the precipitation
data in the dataset. ``mean`` is used when the unit is ``mm/h`` and ``sum``
is used when the unit is ``mm``. For other units an error is raised.
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing
a time series of (ensemble) input fields.
They must be evenly spaced in time.
Expand All @@ -44,7 +53,7 @@ def aggregate_fields_time(dataset: xr.Dataset, time_window_min, ignore_nan=False
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
The new dataset.
See also
Expand Down Expand Up @@ -87,30 +96,37 @@ def aggregate_fields_time(dataset: xr.Dataset, time_window_min, ignore_nan=False
return aggregate_fields(dataset, window_size, dim="time", method=method)


def aggregate_fields_space(dataset: xr.Dataset, space_window, ignore_nan=False):
def aggregate_fields_space(
dataset: xr.Dataset, space_window, ignore_nan=False
) -> xr.Dataset:
"""
Upscale fields in space.
It attempts to aggregate the given dataset in y and x direction in an integer
number of sections of length = ``(window_size_y, window_size_x)``.
If such a aggregation is not possible, an error is raised.
The data is aggregated by computing the mean. Only datasets with precipitation
data in the ``mm`` or ``mm/h`` unit are currently supported.
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing a single field or
a time series of (ensemble) input fields.
space_window: float, tuple or None
The length of the space window that is used to upscale the fields.
If a float is given, the same window size is used for the x- and
y-directions. Separate window sizes are used for x- and y-directions if
a two-element tuple is given. The space_window unit is the same used in
the geographical projection of R and hence the same as for the xpixelsize
and ypixelsize attributes. The space spanned by the n- and m-dimensions
of R must be a multiple of space_window. If set to None, the function
returns a copy of the original R and metadata.
a two-element tuple is given (y, x). The space_window unit is the same
as the unit of x and y in the input dataset. The space spanned by the
n- and m-dimensions of the dataset content must be a multiple of space_window.
If set to None, the function returns a copy of the original dataset.
ignore_nan: bool, optional
If True, ignore nan values.
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
The new dataset.
See also
Expand Down Expand Up @@ -154,18 +170,18 @@ def aggregate_fields_space(dataset: xr.Dataset, space_window, ignore_nan=False):

def aggregate_fields(
dataset: xr.Dataset, window_size, dim="x", method="mean", trim=False
):
) -> xr.Dataset:
"""Aggregate fields along a given direction.
It attempts to aggregate the given R dim in an integer number of sections
It attempts to aggregate the given dataset dim in an integer number of sections
of length = ``window_size``.
If such a aggregation is not possible, an error is raised unless ``trim``
set to True, in which case the dim is trimmed (from the end)
to make it perfectly divisible".
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the input fields.
window_size: int or array-like of ints
The length of the window that is used to aggregate the fields.
Expand Down Expand Up @@ -195,7 +211,7 @@ def aggregate_fields(
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
The new dataset.
See also
Expand Down Expand Up @@ -256,7 +272,7 @@ def clip_domain(dataset: xr.Dataset, extent=None):
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the input fields.
extent: scalars (left, right, bottom, top), optional
The extent of the bounding box in data coordinates to be used to clip
Expand All @@ -268,7 +284,7 @@ def clip_domain(dataset: xr.Dataset, extent=None):
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
The clipped dataset
"""
if extent is None:
Expand Down Expand Up @@ -308,7 +324,7 @@ def square_domain(dataset: xr.Dataset, method="pad", inverse=False):
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the input fields.
method: {'pad', 'crop'}, optional
Either pad or crop.
Expand All @@ -323,7 +339,7 @@ def square_domain(dataset: xr.Dataset, method="pad", inverse=False):
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
the reshaped dataset
"""

Expand Down
16 changes: 8 additions & 8 deletions pysteps/utils/transformation.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def boxcox_transform(
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be transformed.
Lambda: float, optional
Parameter Lambda of the Box-Cox transformation.
Expand All @@ -62,7 +62,7 @@ def boxcox_transform(
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the (back-)transformed units.
References
Expand Down Expand Up @@ -146,7 +146,7 @@ def dB_transform(
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be (back-)transformed.
threshold: float, optional
Optional value that is used for thresholding with the same units as in the dataset.
Expand All @@ -161,7 +161,7 @@ def dB_transform(
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the (back-)transformed units.
"""

Expand Down Expand Up @@ -223,7 +223,7 @@ def NQ_transform(dataset: xr.Dataset, inverse: bool = False, **kwargs) -> xr.Dat
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be transformed.
inverse: bool, optional
If set to True, it performs the inverse transform. False by default.
Expand All @@ -238,7 +238,7 @@ def NQ_transform(dataset: xr.Dataset, inverse: bool = False, **kwargs) -> xr.Dat
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the (back-)transformed units.
References
Expand Down Expand Up @@ -309,14 +309,14 @@ def sqrt_transform(dataset: xr.Dataset, inverse: bool = False, **kwargs) -> xr.D
Parameters
----------
dataset: Dataset
dataset: xarray.Dataset
Dataset to be transformed.
inverse: bool, optional
If set to True, it performs the inverse transform. False by default.
Returns
-------
dataset: Dataset
dataset: xarray.Dataset
Dataset containing the (back-)transformed units.
"""
Expand Down

0 comments on commit 369ef4a

Please sign in to comment.