[PI-3478] Merge cube arithmetic feature branch (SciTools#3785)

* PI-3478: Common metadata API (SciTools#3583) * common metadata api * rationalise _cube_coord_common into common * move state into metadata * MetadataFactory test coverage * temporarily pin back iris-grib * test coverage for iris.common.metadata._BaseMeta * test coverage for iris.common.mixin.LimitedAttributeDict * remove temporary iris-grib pin * review actions * Update lib/iris/tests/unit/common/metadata/test_BaseMetadata.py Co-Authored-By: lbdreyer <[email protected]> * [FB] [PI-3478] Lenient metadata (SciTools#3739) * add lenient infra-structure * add metadata lenient __eq__ support * complete __eq__, combine and difference support * explicit inherited lenient_service + support equal convenience * fix attributes difference + lenient kwargs * make lenient public + minor tidy * rename MetadataManagerFactory to metadata_manager_factory * extend lenient_client decorator to support services registration * add lenient test coverage * purge qualname usage in metadata.py * support global enable for lenient services * support partial mapping metadata assignment * purge Lenient.__setattr__ from api * add BaseMetadata compare test coverage * metadata rationalisation * add BaseMetadata difference test coverage * added context manager ephemeral comment clarification * add BaseMetadata __ne__ test coverage * standardise lenient decorator closure names * add BaseMetadata equal test coverage * half dunder context * add AncillaryVariableMetadata test coverage * add additional AncillaryVariableMetadata test coverage * add CellMeasureMetadata test coverage * Clarify lenient_service operation + simplify code. * add CoordMetadata test coverage * add CubeMetadata test coverage * metadata tests use self.cls * fix typo * fix context manager ephemeral services * add logging * Pin pillow to make graphics tests work again. (SciTools#3630) * Fixed tests since Numpy 1.18 deprecation of non-int num arguments for linspace. (SciTools#3655) * Switched use of datetime.weekday() to datetime.dayofwk. (SciTools#3687) * New image hashes for mpl 3x2 (SciTools#3682) * New image hash for iris.test.test_plot.TestSymbols.test_cloud_cover with matplotlib 3.2.0. * Further images changes for mpl3x2. * Yet more updated image results. * fix sentinel uniqueness test failure * remove redundant cdm mapping test * difference returns None for no difference * protect Lenient and LENIENT private * privitise lenient framework and add API veneer * add explicit maths feature default * review actions * review actions * trexfeathers review actions * stephenworsley review actions Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: Martin Yeo <[email protected]> * [FB] [PI-3478] Lenient cube arithmetic (SciTools#3774) * initial cube arithmetic * support in-place cube resolve * fix non in-place broadcasting * remove temporary resolve scenario test * lenient/strict support for attributes dicts with numpy arrays * lenient/strict treatment of scalar coordinates * strict points/bounds matching * lenient/strict prepare local dim/aux/scalar coordinates * support extended broadcasting * always raise exception on points/bounds mismatch * ignore scalar points/bounds mismatches, lenient only * remove todos * tidy logger debugs * qualify src/tgt cube references in debug * Numpy rounding fix (SciTools#3758) ensure rounding is numpy like (maintains type) * avoid unittest.mock.sentinel copy issue * fast load np.int32 * fix cube maths doctest * fix iris.common.resolve logging configuration * fix prepare points/bounds + extra metadata cml * support mapping reversal based on free dims * var_name fix for lenient equality * add support for DimCoordMetadata * fix circular flag + support CoordMetadata and DimCoordMetadata exchange * fix circular issue for concatenate DimCoord->AuxCoord demotion * fix concatenate._CubeSignature sorted * minor tweaks * keep lenient_client private in maths * tidy maths * tidy iris.analysis.maths.IFunc * refactor IFunc test * polish in-place support * tidy metadata_resolve Co-authored-by: stephenworsley <[email protected]> * rebase master fix-up for cube arithmetic * add missing new dependency to readthedocs.yml requirements Co-authored-by: lbdreyer <[email protected]> Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: Martin Yeo <[email protected]> Co-authored-by: stephenworsley <[email protected]>
tkknight · Aug 14, 2020 · 556e71e · 556e71e
1 parent d16f676
commit 556e71e
Show file tree

Hide file tree

Showing 107 changed files with 11,329 additions and 1,005 deletions.
diff --git a/.flake8 b/.flake8
@@ -15,6 +15,8 @@ ignore =
     E402,
     # E501: line too long
     E501,
+    # E731: do not assign a lambda expression, use a def
+    E731,
     # W503: line break before binary operator
     W503,
     # W504: line break after binary operator
@@ -38,3 +40,7 @@ exclude =
     # ignore third-party files
     #
     gitwash_dumper.py,
+    #
+    # convenience imports
+    #
+    lib/iris/common/__init__.py
diff --git a/ci/requirements/readthedocs.yml b/ci/requirements/readthedocs.yml
@@ -23,6 +23,7 @@ dependencies:
   - netcdf4
   - numpy>=1.14
   - scipy
+  - python-xxhash
 
 # Dependencies needed to run the iris tests
 #------------------------------------------

diff --git a/docs/iris/src/userguide/cube_maths.rst b/docs/iris/src/userguide/cube_maths.rst
@@ -60,6 +60,10 @@ but with the data representing their difference:
          Scalar coordinates:
               forecast_reference_time: 1859-09-01 06:00:00
               height: 1.5 m
+         Attributes:
+              Conventions: CF-1.5
+              Model scenario: E1
+              source: Data from Met Office Unified Model 6.05
 
 
 .. note::

diff --git a/lib/iris/_concatenate.py b/lib/iris/_concatenate.py
@@ -68,7 +68,7 @@ class _CoordMetaData(
     Args:
 
     * defn:
-        The :class:`iris.coords.CoordDefn` metadata that represents a
+        The :class:`iris.common.CoordMetadata` metadata that represents a
         coordinate.
 
     * dims:
@@ -85,7 +85,7 @@ class _CoordMetaData(
 
     """
 
-    def __new__(cls, coord, dims):
+    def __new__(mcs, coord, dims):
         """
         Create a new :class:`_CoordMetaData` instance.
 
@@ -101,7 +101,7 @@ def __new__(cls, coord, dims):
             The new class instance.
 
         """
-        defn = coord._as_defn()
+        defn = coord.metadata
         points_dtype = coord.points.dtype
         bounds_dtype = coord.bounds.dtype if coord.bounds is not None else None
         kwargs = {}
@@ -120,7 +120,7 @@ def __new__(cls, coord, dims):
                 order = _DECREASING
             kwargs["order"] = order
         metadata = super().__new__(
-            cls, defn, dims, points_dtype, bounds_dtype, kwargs
+            mcs, defn, dims, points_dtype, bounds_dtype, kwargs
         )
         return metadata
 
@@ -194,7 +194,7 @@ def __new__(cls, ancil, dims):
             The new class instance.
 
         """
-        defn = ancil._as_defn()
+        defn = ancil.metadata
         metadata = super().__new__(cls, defn, dims)
         return metadata
 
@@ -403,11 +403,11 @@ def __init__(self, cube):
         axes = dict(T=0, Z=1, Y=2, X=3)
 
         # Coordinate sort function - by guessed coordinate axis, then
-        # by coordinate definition, then by dimensions, in ascending order.
+        # by coordinate name, then by dimensions, in ascending order.
         def key_func(coord):
             return (
                 axes.get(guess_coord_axis(coord), len(axes) + 1),
-                coord._as_defn(),
+                coord.name(),
                 cube.coord_dims(coord),
             )
 
@@ -422,7 +422,7 @@ def key_func(coord):
                 self.scalar_coords.append(coord)
 
         def meta_key_func(dm):
-            return (dm._as_defn(), dm.cube_dims(cube))
+            return (dm.metadata, dm.cube_dims(cube))
 
         for cm in sorted(cube.cell_measures(), key=meta_key_func):
             dims = cube.cell_measure_dims(cm)
@@ -990,6 +990,9 @@ def _build_aux_coordinates(self):
                             points, bounds=bnds, **kwargs
                         )
                     except ValueError:
+                        # Ensure to remove the "circular" kwarg, which may be
+                        # present in the defn of a DimCoord being demoted.
+                        _ = kwargs.pop("circular", None)
                         coord = iris.coords.AuxCoord(
                             points, bounds=bnds, **kwargs
                         )

diff --git a/lib/iris/_constraints.py b/lib/iris/_constraints.py
@@ -131,7 +131,7 @@ def _coordless_match(self, cube):
         if self._name:
             # Require to also check against cube.name() for the fallback
             # "unknown" default case, when there is no name metadata available.
-            match = self._name in cube.names or self._name == cube.name()
+            match = self._name in cube._names or self._name == cube.name()
         if match and self._cube_func:
             match = self._cube_func(cube)
         return match

diff --git a/lib/iris/_merge.py b/lib/iris/_merge.py
@@ -22,8 +22,9 @@
     is_lazy_data,
     multidim_lazy_stack,
 )
-import iris.cube
 import iris.coords
+from iris.common import CoordMetadata, CubeMetadata
+import iris.cube
 import iris.exceptions
 import iris.util
 
@@ -115,7 +116,7 @@ class _ScalarCoordPayload(
     Args:
 
     * defns:
-        A list of scalar coordinate definitions :class:`iris.coords.CoordDefn`
+        A list of scalar coordinate metadata :class:`iris.common.CoordMetadata`
         belonging to a :class:`iris.cube.Cube`.
 
     * values:
@@ -1478,9 +1479,7 @@ def axis_and_name(name):
                         )
                     else:
                         bounds = None
-                    kwargs = dict(
-                        zip(iris.coords.CoordDefn._fields, defns[name])
-                    )
+                    kwargs = dict(zip(CoordMetadata._fields, defns[name]))
                     kwargs.update(metadata[name].kwargs)
 
                     def name_in_independents():
@@ -1560,7 +1559,7 @@ def name_in_independents():
                     if bounds is not None:
                         bounds[index] = name_value.bound
 
-                kwargs = dict(zip(iris.coords.CoordDefn._fields, defns[name]))
+                kwargs = dict(zip(CoordMetadata._fields, defns[name]))
                 self._aux_templates.append(
                     _Template(dims, points, bounds, kwargs)
                 )
@@ -1594,7 +1593,7 @@ def _get_cube(self, data):
             (deepcopy(coord), dims)
             for coord, dims in self._aux_coords_and_dims
         ]
-        kwargs = dict(zip(iris.cube.CubeMetadata._fields, signature.defn))
+        kwargs = dict(zip(CubeMetadata._fields, signature.defn))
 
         cms_and_dims = [
             (deepcopy(cm), dims) for cm, dims in self._cell_measures_and_dims
@@ -1794,7 +1793,7 @@ def _extract_coord_payload(self, cube):
 
         # Coordinate sort function.
         # NB. This makes use of two properties which don't end up in
-        # the CoordDefn used by scalar_defns: `coord.points.dtype` and
+        # the metadata used by scalar_defns: `coord.points.dtype` and
         # `type(coord)`.
         def key_func(coord):
             points_dtype = coord.dtype
@@ -1805,14 +1804,14 @@ def key_func(coord):
                 axis_dict.get(
                     iris.util.guess_coord_axis(coord), len(axis_dict) + 1
                 ),
-                coord._as_defn(),
+                coord.metadata,
             )
 
         # Order the coordinates by hints, axis, and definition.
         for coord in sorted(coords, key=key_func):
             if not cube.coord_dims(coord) and coord.shape == (1,):
                 # Extract the scalar coordinate data and metadata.
-                scalar_defns.append(coord._as_defn())
+                scalar_defns.append(coord.metadata)
                 # Because we know there's a single Cell in the
                 # coordinate, it's quicker to roll our own than use
                 # Coord.cell().
@@ -1844,14 +1843,14 @@ def key_func(coord):
 
         factory_defns = []
         for factory in sorted(
-            cube.aux_factories, key=lambda factory: factory._as_defn()
+            cube.aux_factories, key=lambda factory: factory.metadata
         ):
             dependency_defns = []
             dependencies = factory.dependencies
             for key in sorted(dependencies):
                 coord = dependencies[key]
                 if coord is not None:
-                    dependency_defns.append((key, coord._as_defn()))
+                    dependency_defns.append((key, coord.metadata))
             factory_defn = _FactoryDefn(type(factory), dependency_defns)
             factory_defns.append(factory_defn)
 

diff --git a/lib/iris/analysis/__init__.py b/lib/iris/analysis/__init__.py
@@ -319,7 +319,7 @@ def _dimensional_metadata_comparison(*cubes, object_get=None):
                         eq = (
                             other_coord is coord
                             or other_coord.name() == coord.name()
-                            and other_coord._as_defn() == coord._as_defn()
+                            and other_coord.metadata == coord.metadata
                         )
                         if eq:
                             coord_to_add_to_group = other_coord