You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In attempting to compare a GCM and RCM model, I want to regrid the GCM (rectilinear grid) to the RCM grid (rotated pole).
I encountered the following issues:
Iris error message related to RCM cube having both "grid_latitude" and "latitude" coordinates
2024-09-24 18:59:53,984 UTC [2640417] ERROR Failed to run preprocessor function 'regrid' on the data
[<iris 'Cube' of air_temperature / (K) (season_number: 1; latitude: 64; longitude: 128)>]
loaded from original input file(s)
[LocalFile('/space/hall5/sitestore/eccc/crd/crd_shared/ESGF_downloads/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p2f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_historical_r1i1p2f1_gn_185001-201412.nc')]
with function argument(s)
target_grid = Dataset:
{'diagnostic': 'DJF_mean',
'variable_group': 'tas',
'dataset': 'CanRCM5-SN',
'project': 'CORDEX_rja001',
'mip': 'mon',
'short_name': 'tas',
'alias': 'CORDEX_rja001',
'control': True,
'derive': True,
'driver': 'CanESM2',
'ensemble': 'r1i1p1f1',
'exp': 'evaluation',
'force_derivation': False,
'frequency': 'mon',
'institute': 'CCCma',
'long_name': 'Near-Surface Air Temperature',
'modeling_realm': 'atmos',
'original_short_name': 'tas',
'preprocessor': 'DJF_mean_tas',
'recipe_dataset_index': 1,
'runid': 'NAM-50',
'standard_name': 'air_temperature',
'timerange': '2014/2014',
'units': 'K'}
session: 'TEST_TEST_regrid_RCM_20240924_185914',
scheme = {'reference': 'esmvalcore.preprocessor.regrid_schemes:ESMPyNearest'}
2024-09-24 18:59:55,195 UTC [2632345] ERROR Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 816, in _run_task
output_files = task.run()
^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 264, in run
self.output_files = self._run(input_files)
^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 684, in _run
product.apply(step, self.debug)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 492, in apply
self.cubes = preprocess(self.cubes, step,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 401, in preprocess
result.append(_run_preproc_function(function, item, settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 346, in _run_preproc_function
return function(items, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_shared.py", line 262, in wrapper
result = func(data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_regrid.py", line 848, in regrid
cube = regridder(cube)
^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_regrid_esmpy.py", line 98, in __call__
result = map_slices(cube, regridder, src_rep, dst_rep)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_mapping.py", line 224, in map_slices
aux_coords_and_dims += [(c, src.coord_dims(c)) for c in dst_rep.aux_coords]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_mapping.py", line 224, in <listcomp>
aux_coords_and_dims += [(c, src.coord_dims(c)) for c in dst_rep.aux_coords]
^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/iris/cube.py", line 1894, in coord_dims
coord = self.coord(coord)
^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/iris/cube.py", line 2298, in coord
raise iris.exceptions.CoordinateNotFoundError(emsg)
iris.exceptions.CoordinateNotFoundError: "Expected to find exactly 1 coordinate matching the given 'latitude' coordinate's metadata, but found none."
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 533, in run
fire.Fire(ESMValTool())
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 413, in run
self._run(recipe, session)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 455, in _run
process_recipe(recipe_file=recipe, session=session)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 130, in process_recipe
recipe.run()
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_recipe/recipe.py", line 1095, in run
self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 738, in run
self._run_parallel(address, max_parallel_tasks)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 782, in _run_parallel
_copy_results(task, running[task])
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 805, in _copy_results
task.output_files, task.products = future.get()
^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
iris.exceptions.CoordinateNotFoundError: "Expected to find exactly 1 coordinate matching the given 'latitude' coordinate's metadata, but found none."
I resolved this by editing esmvalcore.preprocessor._regrid.py to use from iris.analysis.cartography import _get_lon_lat_coords, which prioritizes "grid_<coord>" over "<coord>":
A second issue was that the auxialliary coordinates differed between the models. Looking at the intermediate saved cubes between preprocessing steps, at the cubes saved prior to attempting regridding:
This caused an issue in comparing the aux_coords_and_dims in esmvalcore/preprocessor/_mapping.py.
An inelegant fix was to add a try/except clause:
diff --git a/esmvalcore/preprocessor/_mapping.py b/esmvalcore/preprocessor/_mapping.py
index 28d1fefb8..ccd8dc598 100644
--- a/esmvalcore/preprocessor/_mapping.py
+++ b/esmvalcore/preprocessor/_mapping.py
@@ -221,7 +221,10 @@ def map_slices(src, func, src_rep, dst_rep):
dim_coords = src_keep_spec[1] + dst_rep.coords(dim_coords=True)
dim_coords_and_dims = [(c, i) for i, c in enumerate(dim_coords)]
aux_coords_and_dims = [(c, src.coord_dims(c)) for c in src_keep_spec[2]]
- aux_coords_and_dims += [(c, src.coord_dims(c)) for c in dst_rep.aux_coords]
+ try:
+ aux_coords_and_dims += [(c, src.coord_dims(c)) for c in dst_rep.aux_coords]
+ except:
+ pass
A related unresolved issue I've encountered when trying to use the iris regridder (i.e., chosen when setting scheme: linear).
Both GCM and RCM datasets have a coordinate system specified. I can successfully do the reverse regridding (RCM-->GCM).
2024-09-24 19:10:06,547 UTC [2683224] ERROR Failed to run preprocessor function 'regrid' on the data
[<iris 'Cube' of air_temperature / (K) (season_number: 1; latitude: 64; longitude: 128)>]
loaded from original input file(s)
[LocalFile('/space/hall5/sitestore/eccc/crd/crd_shared/ESGF_downloads/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p2f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_historical_r1i1p2f1_gn_185001-201412.nc')]
with function argument(s)
target_grid = Dataset:
{'diagnostic': 'DJF_mean',
'variable_group': 'tas',
'dataset': 'CanRCM5-SN',
'project': 'CORDEX_rja001',
'mip': 'mon',
'short_name': 'tas',
'alias': 'CORDEX_rja001',
'control': True,
'derive': True,
'driver': 'CanESM2',
'ensemble': 'r1i1p1f1',
'exp': 'evaluation',
'force_derivation': False,
'frequency': 'mon',
'institute': 'CCCma',
'long_name': 'Near-Surface Air Temperature',
'modeling_realm': 'atmos',
'original_short_name': 'tas',
'preprocessor': 'DJF_mean_tas',
'recipe_dataset_index': 1,
'runid': 'NAM-50',
'standard_name': 'air_temperature',
'timerange': '2014/2014',
'units': 'K'}
session: 'TEST_TEST_regrid_RCM_20240924_190952',
scheme = 'linear'
2024-09-24 19:10:07,494 UTC [2673822] ERROR Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 816, in _run_task
output_files = task.run()
^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 264, in run
self.output_files = self._run(input_files)
^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 703, in _run
product.apply(step, self.debug)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 511, in apply
self.cubes = preprocess(self.cubes, step,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 420, in preprocess
result.append(_run_preproc_function(function, item, settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 365, in _run_preproc_function
return function(items, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_shared.py", line 262, in wrapper
result = func(data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_regrid.py", line 848, in regrid
cube = regridder(cube)
^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/iris/analysis/_regrid.py", line 919, in __call__
raise ValueError(
ValueError: The rectilinear grid coordinates of the given cube and target grid must either both have coordinate systems or both have no coordinate system but with matching coordinate metadata.
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 533, in run
fire.Fire(ESMValTool())
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 413, in run
self._run(recipe, session)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 455, in _run
process_recipe(recipe_file=recipe, session=session)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_main.py", line 130, in process_recipe
recipe.run()
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_recipe/recipe.py", line 1096, in run
self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 738, in run
self._run_parallel(address, max_parallel_tasks)
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 782, in _run_parallel
_copy_results(task, running[task])
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 805, in _copy_results
task.output_files, task.products = future.get()
^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 816, in _run_task
output_files = task.run()
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/_task.py", line 264, in run
self.output_files = self._run(input_files)
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 703, in _run
product.apply(step, self.debug)
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 511, in apply
self.cubes = preprocess(self.cubes, step,
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 420, in preprocess
result.append(_run_preproc_function(function, item, settings,
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/__init__.py", line 365, in _run_preproc_function
return function(items, **kwargs)
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_shared.py", line 262, in wrapper
result = func(data, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/fs/site5/eccc/crd/ccrn/users/rkw001/code/esmvalcore/ESMValCore_v2.11.0/esmvalcore/preprocessor/_regrid.py", line 848, in regrid
cube = regridder(cube)
^^^^^^^^^^^^^^^^^
File "/space/hall5/sitestore/eccc/crd/ccrn/users/rkw001/miniconda3/envs/a4d_env_v1.5/lib/python3.11/site-packages/iris/analysis/_regrid.py", line 919, in __call__
raise ValueError(
ValueError: The rectilinear grid coordinates of the given cube and target grid must either both have coordinate systems or both have no coordinate system but with matching coordinate metadata.
The text was updated successfully, but these errors were encountered:
Using ESMVal* tagged releases v2.11.0
In attempting to compare a GCM and RCM model, I want to regrid the GCM (rectilinear grid) to the RCM grid (rotated pole).
I encountered the following issues:
Iris error message related to RCM cube having both "grid_latitude" and "latitude" coordinates
I resolved this by editing
esmvalcore.preprocessor._regrid.py
to usefrom iris.analysis.cartography import _get_lon_lat_coords
, which prioritizes "grid_<coord>" over "<coord>":A second issue was that the auxialliary coordinates differed between the models. Looking at the intermediate saved cubes between preprocessing steps, at the cubes saved prior to attempting regridding:
GCM cube coords:
RCM cube coords:
This caused an issue in comparing the
aux_coords_and_dims
inesmvalcore/preprocessor/_mapping.py
.An inelegant fix was to add a try/except clause:
Recipe:
A related unresolved issue I've encountered when trying to use the iris regridder (i.e., chosen when setting
scheme: linear
).Both GCM and RCM datasets have a coordinate system specified. I can successfully do the reverse regridding (RCM-->GCM).
The text was updated successfully, but these errors were encountered: