Derived coordinates are not loaded anymore? #3961

valeriupredoi · 2021-01-26T17:28:36Z

📰 Custom Issue

Hey guys, one more for me today, I am using this file and loading it simple:

import iris

cs = iris.load("/home/valeriu/ESMValCore/tests/integration/cmor/_fixes/test_data/common_cl_ap.nc")
for c in cs:
    print(c)

with iris=3 I am not seeing the derived coordinates no more:

with iris=2.4:

b_bnds / (1)                        (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
     Dimension coordinates:
          atmosphere_hybrid_sigma_pressure_coordinate                           x       -
     Auxiliary coordinates:
          ap                                                                    x       -
          b                                                                     x       -
     Derived coordinates:
          air_pressure                                                          x       -
ap_bnds / (1)                       (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
     Dimension coordinates:
          atmosphere_hybrid_sigma_pressure_coordinate                           x       -
     Auxiliary coordinates:
          ap                                                                    x       -
          b                                                                     x       -
     Derived coordinates:
          air_pressure                                                          x       -
cloud_area_fraction_in_atmosphere_layer    (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
     Dimension coordinates:
          time                                            x                                               -            -             -
          atmosphere_hybrid_sigma_pressure_coordinate     -                                               x            -             -
          latitude                                        -                                               -            x             -
          longitude                                       -                                               -            -             x
     Auxiliary coordinates:
          surface_air_pressure                            x                                               -            x             x
          ap                                              -                                               x            -             -
          b                                               -                                               x            -             -
     Derived coordinates:
          air_pressure                                    x                                               x            x             x
surface_air_pressure / (Pa)         (time: 1; latitude: 3; longitude: 4)
     Dimension coordinates:
          time                           x            -             -
          latitude                       -            x             -
          longitude                      -            -             x
     Attributes:
          additional_attribute: xyz

with iris=3.0:

b_bnds / (unknown)                  (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
     Dimension coordinates:
          atmosphere_hybrid_sigma_pressure_coordinate                           x       -
     Auxiliary coordinates:
          ap                                                                    x       -
          b                                                                     x       -
ap_bnds / (unknown)                 (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
     Dimension coordinates:
          atmosphere_hybrid_sigma_pressure_coordinate                           x       -
     Auxiliary coordinates:
          ap                                                                    x       -
          b                                                                     x       -
cloud_area_fraction_in_atmosphere_layer       (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
     Dimension coordinates:
          time                                            x                                               -            -             -
          atmosphere_hybrid_sigma_pressure_coordinate     -                                               x            -             -
          latitude                                        -                                               -            x             -
          longitude                                       -                                               -            -             x
     Auxiliary coordinates:
          surface_air_pressure                            x                                               -            x             x
          ap                                              -                                               x            -             -
          b                                               -                                               x            -             -
surface_air_pressure / (Pa)         (time: 1; latitude: 3; longitude: 4)
     Dimension coordinates:
          time                           x            -             -
          latitude                       -            x             -
          longitude                      -            -             x
     Attributes:
          additional_attribute: xyz

where did the Derived coordinates go?

The text was updated successfully, but these errors were encountered:

bjlittle · 2021-01-26T22:31:36Z

@valeriupredoi Ohh very interesting, I'll take a peek for you...

bjlittle · 2021-01-27T01:44:16Z

@valeriupredoi Okay, so this took a wee bit of digging, but the mystery is thankfully solved 🕵️

Firstly, I can recreate your issue in iris 3.0.0, and I can also confirm that iris 2.4.0 does indeed realise derived coordinates for your example dataset 👍

This change in behaviour between iris 3.0.0 and 2.4.0 directly relates to PR #3795, where the units of an AuxCood or DimCoord that is not explicitly specified defaults to unknown rather then 1 (dimensionless).

We thought about this change in behaviour long and hard, as the treatment of default units was inconsistent across iris for different objects, and the CF Metadata standard did not add any clarity (no surprises there, indeed it was inconsistent), but in the end we unified the behaviour for consistency sake. That is, erring on the side that it correct and safer to assign units as unknown in the case when they are not specified, whereas it is potentially incorrect and not safe to assume default dimensionless units instead.

This change in behaviour is mentioned in the iris 3.0.0 release notes in the Incompatible Changes section, fifth bullet point.

Where this is stinging you is for the missing units for the sigma variable associated with the atmosphere_hybrid_sigma_pressure_coordinate i.e., given formula_terms = "ap: ap_bnds b: b_bnds ps: ps", sigma in this case is the NetCDF b variable, and this is defined within the example file simply as double b(lev) ;

As the b variable has no units associated with it iris 3.0.0 assigns the default units of unknown (which they are). However, during the creation of the auxiliary factory that creates the derived coordinate it enforces the contract that the sigma variable must be dimensionless, which in this case it's not, as it's unknown. Note that, enforcing that sigma is dimensionless has not changed from 2.4.0 to 3.0.0.

This failure for sigma units results in iris raising a warning and not creating the associated HybridPressureFactory auxiliary factory i.e., during loading you see the following warning:

UserWarning: Invalid units: sigma must be dimensionless.

Also, note the example given in the CF Metadata Conventions, where their sigma variable has explicit units = "1".

So in order to fix this, my advise is to add the correct units to your sigma variable in the file. In addition to this, you could also add the missing bounds attribute for the sigma (b) and delta (ap) variables e.g.,

netcdf common_cl_ap {
dimensions:
	time = 1 ;
	lev = 2 ;
	lat = 3 ;
	lon = 4 ;
	bnds = 2 ;
variables:
	double time(time) ;
		time:standard_name = "time" ;
		time:units = "days since 6543-2-1" ;
	double lev(lev) ;
		lev:bounds = "lev_bnds" ;
		lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
		lev:units = "1" ;
		lev:formula_terms = "ap: ap b: b ps: ps" ;
	double lev_bnds(lev, bnds) ;
		lev_bnds:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
		lev_bnds:units = "1" ;
		lev_bnds:formula_terms = "ap: ap_bnds b: b_bnds ps: ps" ;
	double lat(lat) ;
		lat:standard_name = "latitude" ;
		lat:units = "degrees_north" ;
	double lon(lon) ;
		lon:standard_name = "longitude" ;
		lon:units = "degrees_east" ;
	double b(lev) ;
                b:units = "1" ;                                                 <== add "units" for sigma
                b:bounds = "b_bnds" ;                                           <== add "bounds" for sigma
	double b_bnds(lev, bnds) ;
	double ps(time, lat, lon) ;
		ps:standard_name = "surface_air_pressure" ;
		ps:units = "Pa" ;
		ps:additional_attribute = "xyz" ;
	float cl(time, lev, lat, lon) ;
		cl:standard_name = "cloud_area_fraction_in_atmosphere_layer" ;
		cl:units = "%" ;
	double ap(lev) ;
		ap:units = "Pa" ;
                ap:bounds = "ap_bnds" ;                                         <== add "bounds" for delta
	double ap_bnds(lev, bnds) ;
}

Hope this helps 😄

bjlittle · 2021-01-27T07:49:47Z

The alternative is that we relax the contract in the auxiliary factories to accept coordinates that are dimensionless or unknown. However that could cause other undesirable side effects or complications e.g., The computation of the derived coordinate would break due to the use of sigma with unknown units (I think) or at least you'd end up with a derived coordinate also with units of unknown.

However, iris could override the unknown units to be dimensionless. This may affect load/save round-tripping... that would need to be investigated. Regardless, this would be an iris 3.0.1 release

bjlittle · 2021-01-27T09:21:06Z

@valeriupredoi Done a quick hack to see, if in principle, this promotion of units of unknown to dimensionless (1) within auxiliary factories works...

Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.12.0
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15) 
[GCC 7.5.0] on linux
>>> import iris
>>> iris.__version__
'3.0.0'
>>> fname = "/downloads/common_cl_ap.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'ap_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)
>>> print(cubes)
0: ap_bnds / (unknown)                 (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
1: b_bnds / (unknown)                  (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
2: cloud_area_fraction_in_atmosphere_layer / (%) (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
3: surface_air_pressure / (Pa)         (time: 1; latitude: 3; longitude: 4)
print(cubes[2])
cloud_area_fraction_in_atmosphere_layer    (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
     Dimension coordinates:
          time                                            x                                               -            -             -
          atmosphere_hybrid_sigma_pressure_coordinate     -                                               x            -             -
          latitude                                        -                                               -            x             -
          longitude                                       -                                               -            -             x
     Auxiliary coordinates:
          surface_air_pressure                            x                                               -            x             x
          ap                                              -                                               x            -             -
          b                                               -                                               x            -             -
     Derived coordinates:
          air_pressure                                    x                                               x            x             x

seems possible

bjlittle · 2021-01-27T09:23:29Z

@bouweandela and @valeriupredoi How much of a deal breaker is this for your guys? I'm assuming an iris 3.0.1 would be preferable?

valeriupredoi · 2021-01-27T09:41:49Z

@bjlittle cheers for the super quick and detailed answers, mate! We appreciate very much your very quick response times to issues we raise 🍺 About this, I am afraid it generates a chicken-and-egg problem - the chicken being faulty model data that is already on the ESGF and the egg is us needing to fix it via our CMOR fixes before we put it through the analysis. The file I linked above is a replication of such a faulty model data file that we use in a test of a fix: if we can't load the derived coordinates it means we can't fix them in ESMValCore and that means we will not be able to run ESMValTool on it, well, at least without not turning off all the CMOR checks that, in turn, will generate a whole lot of other headaches. So yeah, I think this is something we kind of really need 😁

bjlittle · 2021-01-27T09:46:20Z

@valeriupredoi Okay, no worries dude. Leave it with me and I'll get back to you asap... just need to speak to the team about this 👍

valeriupredoi · 2021-01-27T09:47:37Z

@valeriupredoi Okay, no worries dude. Leave it with me and I'll get back to you asap... just need to speak to the team about this +1

you're legend @bjlittle cheers muchly! 🍺

bjlittle · 2021-01-27T10:12:02Z

@valeriupredoi Okay, I'm dropping everything today to service this - I can totally appreciate the situation you're in, so I'm going to push an iris 3.0.1 release today.

Hopefully, I can turn this around quickly for you, as I know you're working to a tight deadline.

I'll ping you here @valeriupredoi with my progress on this to keep you in the loop 👍

valeriupredoi · 2021-01-27T10:29:32Z

@bjlittle that's fantastic, sorry to have done this to you 😁 We are planning on frezing ESMValCore on February 1st, as per our release schedule, but the freeze lasts a week and in extraordinary circumstances we can still push to main during that week - so - this would be good to be in so we can push iris=3 in the release, but that means it's not a race to the sea kind of thing. @jvegasbsc is the release manager this time around - heads up, Javi 👍 Cheers muchly for your efforts, Bill and team! 🍺

bouweandela · 2021-01-27T11:01:50Z

@bouweandela and @valeriupredoi How much of a deal breaker is this for your guys? I'm assuming an iris 3.0.1 would be preferable?

@schlunma Could you comment on this?

schlunma · 2021-01-27T11:14:16Z

From a quick look at the issue I'm pretty sure that we can easily work around this issue in our fixes by setting the units to '1' like @bjlittle suggested (we actually do that in one example). I'll can have a detailed look at this later in the afternoon today.

By the way I really like the new behavior of iris (using unknown as units if no unit is specified), since I had troubles with the old behavior before 👍

valeriupredoi · 2021-01-27T11:16:55Z

From a quick look at the issue I'm pretty sure that we can easily work around this issue in our fixes by setting the units to '1' like @bjlittle suggested (we actually do that in one example). I'll can have a detailed look at this later in the afternoon today.

yes, but the coordinate is not loaded at load point!

schlunma · 2021-01-27T11:20:16Z

The particular file here is so messed up that we have to fix it with netCDF4 anyway, so we can do it with that.

Nevertheless, even though the derived coordinate is not there we should be able to add it after loading with iris.cube.Cube.add_aux_factory.

valeriupredoi · 2021-01-27T11:22:55Z

how about the case where the file is a model file off the ESGF and needs fixes - this very case that we test? Don't assume any level of clean data, you have seen what messed up files can still be out there on ESGF

bjlittle · 2021-01-27T11:26:44Z

@schlunma seems like a reasonable work around, but it's really causing you guys quite a lot of extra work to jump through hoops.

I've got a fix that I'm testing just now that would gracefully promote any formula terms variable that must be dimensionless to be dimensionless if it has units of unknown - it's doing this within the aux factory itself.

Also, this behavioural change applies to 6 aux factories - so for me, it makes more sense to fix this in iris rather than force this upon all users.

I'm going to roll with this fix regardlessly, as my expectation is that this will affect many users.

Question: do you have any other ESMValTool netCDF files that also show this failure that I can test my fix with?

bjlittle · 2021-01-27T11:28:23Z

If this fix is good, and there are no other side-effects, then I believe that i can push an iris 3.0.1 release today and get it on conda-forge for you guys.

I have @trexfeathers on hand as a review buddy to expedite the review and release.

valeriupredoi · 2021-01-27T11:28:35Z

@bjlittle hang on, I'm rerunning the tests, try find more files, if @schlunma doesn't beat me to it :)

schlunma · 2021-01-27T11:36:24Z

@bjlittle Thanks, that sounds like a great solution!

bjlittle · 2021-01-27T13:57:52Z

I've got one file, the other is almost there... so glad we did this as the file cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412 contains the alternative formula_terms = "p0: p0 a: a b: b ps: ps" for a atmosphere_hybrid_sigma_pressure_coordinate, where ap (delta) is formed by performing p0 * a.

In the example file a has no units and p0 has units of Pa (which does and must match ps), however the p0 * a results in a delta with units of unknown and causes the same failure, but in a different way.

I've located the fix in iris/fileformats/netcdf.py before it then creates the aux factory, as it's too late for the patch in the aux factory to fix this particular nuance.

Re-testing now...

bjlittle · 2021-01-27T14:06:37Z

Tis a beautiful thing...

>>> import iris
>>> fname = "/downloads/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
  message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'a_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)

>>> cubes
[<iris 'Cube' of vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)>,
<iris 'Cube' of vertical coordinate formula term: a(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)>,
<iris 'Cube' of Surface Air Pressure / (Pa) (time: 1980; latitude: 64; longitude: 128)>,
<iris 'Cube' of cloud_area_fraction_in_atmosphere_layer / (%) (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)>]

>>> print(cubes[3])
cloud_area_fraction_in_atmosphere_layer             (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)
     Dimension coordinates:
          time                                                     x                                                  -             -              -
          atmosphere_hybrid_sigma_pressure_coordinate              -                                                  x             -              -
          latitude                                                 -                                                  -             x              -
          longitude                                                -                                                  -             -              x
     Auxiliary coordinates:
          Surface Air Pressure                                     x                                                  -             x              x
          vertical coordinate formula term: a(k)                   -                                                  x             -              -
          vertical coordinate formula term: b(k)                   -                                                  x             -              -
          vertical pressure                                        -                                                  x             -              -
     Derived coordinates:
          air_pressure                                             x                                                  x             x              x
     Scalar coordinates:
          vertical coordinate formula term: reference pressure: 100000.0 Pa
     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: CMIP
          branch_method: Standard
          branch_time_in_child: 0.0
          branch_time_in_parent: 2110.0
          cmor_version: 3.3.2
          comment: Percentage cloud cover, including both large-scale and convective clou...
          contact: Dr. Tongwen Wu(twwu@cma.gov.cn)
          creation_date: 2018-12-17T02:20:59Z
          data_specs_version: 01.00.27
          description: DECK: historical
          experiment: all-forcing simulation of the recent past
          experiment_id: historical
          external_variables: areacella
          forcing_index: 1
          frequency: mon
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.BCC.BCC-ESM1.historical.none.r1i1...
          grid: T42
          grid_label: gn
          history: 2018-12-17T02:20:59Z altered by CMOR: Inverted axis: lev.
          initialization_index: 1
          institution: Beijing Climate Center, Beijing 100081, China
          institution_id: BCC
          license: CMIP6 model data produced by BCC is licensed under a Creative Commons Attribution...
          mip_era: CMIP6
          nominal_resolution: 250 km
          original_name: CLOUD
          parent_activity_id: CMIP
          parent_experiment_id: piControl
          parent_mip_era: CMIP6
          parent_source_id: BCC-ESM1
          parent_time_units: days since 1850-01-01
          parent_variant_label: r1i1p1f1
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: atmos
          references: Model described by Tongwen Wu et al. (JGR 2013; JMR 2014; submmitted to...
          run_variant: forcing: greenhouse gases,aerosol emission,solar constant,volcano mass,land...
          source: BCC-ESM 1 (2017):   aerosol: none  atmos: BCC_AGCM3_LR (T42; 128 x 64 longitude/latitude;...
          source_id: BCC-ESM1
          source_type: AER AOGCM CHEM
          sub_experiment: none
          sub_experiment_id: none
          table_id: Amon
          table_info: Creation Date:(30 July 2018) MD5:e53ff52009d0b97d9d867dc12b6096c7
          title: BCC-ESM1 output prepared for CMIP6
          tracking_id: hdl:21.14100/6f5b873f-9609-4be8-99d6-bfde65468e67
          variable_id: cl
          variant_label: r1i1p1f1
     Cell methods:
          mean: area (20 minutes), time (20 minutes)

bjlittle · 2021-01-27T14:08:42Z

Just checking the second file cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc...

bjlittle · 2021-01-27T14:10:53Z

Sweet, this looks good too...

>>> fname2 = "/downloads/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
  message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'a_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
  "{!r}".format(name)

>>> print(cubes)
0: vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)
1: vertical coordinate formula term: a(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)
2: Surface Air Pressure / (Pa)         (time: 1980; latitude: 64; longitude: 128)
3: cloud_area_fraction_in_atmosphere_layer / (%) (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)

>>> print(cubes[3])
cloud_area_fraction_in_atmosphere_layer             (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)
     Dimension coordinates:
          time                                                     x                                                  -             -              -
          atmosphere_hybrid_sigma_pressure_coordinate              -                                                  x             -              -
          latitude                                                 -                                                  -             x              -
          longitude                                                -                                                  -             -              x
     Auxiliary coordinates:
          Surface Air Pressure                                     x                                                  -             x              x
          vertical coordinate formula term: a(k)                   -                                                  x             -              -
          vertical coordinate formula term: b(k)                   -                                                  x             -              -
          vertical pressure                                        -                                                  x             -              -
     Derived coordinates:
          air_pressure                                             x                                                  x             x              x
     Scalar coordinates:
          vertical coordinate formula term: reference pressure: 100000.0 Pa
     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: CMIP
          branch_method: Standard
          branch_time_in_child: 0.0
          branch_time_in_parent: 2110.0
          cmor_version: 3.3.2
          comment: Percentage cloud cover, including both large-scale and convective clou...
          contact: Dr. Tongwen Wu(twwu@cma.gov.cn)
          creation_date: 2018-12-17T02:20:59Z
          data_specs_version: 01.00.27
          description: DECK: historical
          experiment: all-forcing simulation of the recent past
          experiment_id: historical
          external_variables: areacella
          forcing_index: 1
          frequency: mon
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.BCC.BCC-ESM1.historical.none.r1i1...
          grid: T42
          grid_label: gn
          history: 2018-12-17T02:20:59Z altered by CMOR: Inverted axis: lev.
          initialization_index: 1
          institution: Beijing Climate Center, Beijing 100081, China
          institution_id: BCC
          license: CMIP6 model data produced by BCC is licensed under a Creative Commons Attribution...
          mip_era: CMIP6
          nominal_resolution: 250 km
          original_name: CLOUD
          parent_activity_id: CMIP
          parent_experiment_id: piControl
          parent_mip_era: CMIP6
          parent_source_id: BCC-ESM1
          parent_time_units: days since 1850-01-01
          parent_variant_label: r1i1p1f1
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: atmos
          references: Model described by Tongwen Wu et al. (JGR 2013; JMR 2014; submmitted to...
          run_variant: forcing: greenhouse gases,aerosol emission,solar constant,volcano mass,land...
          source: BCC-ESM 1 (2017):   aerosol: none  atmos: BCC_AGCM3_LR (T42; 128 x 64 longitude/latitude;...
          source_id: BCC-ESM1
          source_type: AER AOGCM CHEM
          sub_experiment: none
          sub_experiment_id: none
          table_id: Amon
          table_info: Creation Date:(30 July 2018) MD5:e53ff52009d0b97d9d867dc12b6096c7
          title: BCC-ESM1 output prepared for CMIP6
          tracking_id: hdl:21.14100/6f5b873f-9609-4be8-99d6-bfde65468e67
          variable_id: cl
          variant_label: r1i1p1f1
     Cell methods:
          mean: area (20 minutes), time (20 minutes)

valeriupredoi · 2021-01-27T14:11:27Z

     Derived coordinates:
          air_pressure                                             x

yay! welcome to the absolute nightmare on ESGF street @bjlittle 🤣

bjlittle · 2021-01-27T14:15:50Z

Okay, I think we're good guys 👍

I defo need to add some unit test coverage for these changes, but that shouldn't take too long.

I've already updated the docs for the v3.0.1 patch release, bumped the version et al.

Once I nail the tests, I'll try to expedite this through review. Tag a release, and the conda-forge bot should automagically create a PR on the feedstock. So iris 3.0.1 should be up on conda-forge in a couple of hours from now 🤞

If any gotchas happen between now and then I'll let you know here asap

schlunma · 2021-01-27T14:17:02Z

The UKESM model should have an altitude derived coordinate, that's weird!

EDIT: You need to use fname2 instead of fname in the beginning 😄

valeriupredoi · 2021-01-27T14:18:13Z

that's terrific! Thanks very much @bjlittle - a very nice example of cross-package collaboration when you have cool devvs on both sides, remind me to buy you a 🍺 when all this mess is over (the pandemic not the ESGF data issues, the latter one won't be over soon) 😁

bjlittle · 2021-01-27T14:18:20Z

LOL what a muppet... hang tight...

bjlittle · 2021-01-27T14:20:58Z

Great spot @schlunma, thanks... here we go:

>>> fname2 = "/downloads/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc"
>>> cubes = iris.load(fname2)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
  message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'orog' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('lat', 'lon') do not span ('lev', 'bnds')
  warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'orog'
  "{!r}".format(name)

>>> print(cubes)
0: vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_height_coordinate: 85; -- : 2)
1: Surface Altitude / (m)              (latitude: 144; longitude: 192)
2: cloud_area_fraction_in_atmosphere_layer / (%) (time: 600; atmosphere_hybrid_height_coordinate: 85; latitude: 144; longitude: 192)

>>> print(cubes[2])
cloud_area_fraction_in_atmosphere_layer / (%) (time: 600; atmosphere_hybrid_height_coordinate: 85; latitude: 144; longitude: 192)
     Dimension coordinates:
          time                                     x                                         -             -               -
          atmosphere_hybrid_height_coordinate      -                                         x             -               -
          latitude                                 -                                         -             x               -
          longitude                                -                                         -             -               x
     Auxiliary coordinates:
          vertical coordinate formula term: b(k)   -                                         x             -               -
          Surface Altitude                         -                                         -             x               x
     Derived coordinates:
          altitude                                 -                                         x             x               x
     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: CMIP
          branch_method: standard
          branch_time_in_child: 0.0
          branch_time_in_parent: 144000.0
          cmor_version: 3.4.0
          comment: Percentage cloud cover, including both large-scale and convective clou...
          creation_date: 2019-04-05T15:57:16Z
          cv_version: 6.2.20.1
          data_specs_version: 01.00.29
          experiment: all-forcing simulation of the recent past
          experiment_id: historical
          external_variables: areacella
          forcing_index: 2
          frequency: mon
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.MOHC.UKESM1-0-LL.historical.none....
          grid: Native N96 grid; 192 x 144 longitude/latitude
          grid_label: gn
          history: 2019-04-05T15:57:16Z altered by CMOR: Converted units from '1' to '%'....
          initialization_index: 1
          institution: Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK
          institution_id: MOHC
          license: CMIP6 model data produced by the Met Office Hadley Centre is licensed under...
          mip_era: CMIP6
          mo_runid: u-bc179
          nominal_resolution: 250 km
          original_name: mo: (stash: m01s02i261, lbproc: 128)
          original_units: 1
          parent_activity_id: CMIP
          parent_experiment_id: piControl
          parent_mip_era: CMIP6
          parent_source_id: UKESM1-0-LL
          parent_time_units: days since 1850-01-01-00-00-00
          parent_variant_label: r1i1p1f2
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: atmos
          source: UKESM1.0-LL (2018): 
aerosol: UKCA-GLOMAP-mode
atmos: MetUM-HadGEM3-GA7.1...
          source_id: UKESM1-0-LL
          source_type: AOGCM AER BGC CHEM
          sub_experiment: none
          sub_experiment_id: none
          table_id: Amon
          table_info: Creation Date:(13 December 2018) MD5:2b12b5db6db112aa8b8b0d6c1645b121
          title: UKESM1-0-LL output prepared for CMIP6
          tracking_id: hdl:21.14100/7507e315-b264-43ff-a7c6-479946c0a033
          variable_id: cl
          variant_label: r1i1p1f2
     Cell methods:
          mean: area, time

valeriupredoi · 2021-01-27T14:21:15Z

do you still need my chopped files on GitHub @bjlittle - or I can remove them? Might get Github to banish me for uploading junk 😁

bjlittle · 2021-01-27T14:21:24Z

You guys happy with this?

bjlittle · 2021-01-27T14:21:56Z

@valeriupredoi thanks, nuke the repo dude 👍

valeriupredoi · 2021-01-27T14:22:44Z

You guys happy with this?

looks good to me but @schlunma is the hybrid guru ->

bjlittle · 2021-01-27T14:30:11Z

I've checked all the other aux factories that iris implements, and cross referenced with the CF Metadata Conventions, Appendix D: Parametric Vertical Coordinates and there isn't another example of a parametric vertical coordinate with alternative formulas to derive the same quantity, apart from Atmosphere hybrid sigma pressure coordinate.

So I don't think there are other similar suprises lurking in the depths waiting to surprise us...

bjlittle · 2021-01-27T14:31:10Z

Unless you tell me otherwise, I'm cracking on with 3.0.1...

valeriupredoi · 2021-01-27T14:33:22Z

schlunma · 2021-01-27T14:47:19Z

Cheers @bjlittle, I'm happy with it 👍 Thanks for your quick help!!

trexfeathers · 2021-01-27T16:46:56Z

Closed by #3965

bjlittle · 2021-01-27T18:44:25Z

@bouweandela @valeriupredoi @schlunma @jvegasbsc

iris 3.0.1 has already been tagged, the docs have already been built on readthedocs, and I've just kicked of an update on the conda-forge iris-feedstock... almost there 😉

I'll ping the ESMValTool WhatsApp thread when conda-forge has pushed the built iris 3.0.1 package to the Anaconda Cloud channel - then you can install and test. Good luck! 🤞

valeriupredoi · 2021-01-27T19:01:08Z

legend @bjlittle cheers very much! 🍺

bouweandela · 2021-01-27T19:36:39Z

Thanks for the amazingly quick bugfix and release! 🥳

bjlittle · 2021-01-27T20:16:01Z

Hit conda search -c conda-forge iris until iris 3.0.1 pops up... not sure when the channel is populated

bjlittle · 2021-01-27T20:19:28Z

💥 3.0.1 is live on conda-forge! go for it!

bjlittle · 2021-01-27T20:22:14Z

Installed iris 3.0.1 and tested... seems to work for me 😉

valeriupredoi added the New: Issue label Jan 26, 2021

valeriupredoi mentioned this issue Jan 26, 2021

Switch to Iris 3 ESMValGroup/ESMValCore#819

Merged

9 tasks

bjlittle self-assigned this Jan 26, 2021

bjlittle added Status: Won't Fix Status: Decision Required and removed Status: Won't Fix labels Jan 27, 2021

bjlittle added Status: Work in Progress and removed Status: Decision Required labels Jan 27, 2021

bjlittle added this to the v3.0.0 milestone Jan 27, 2021

valeriupredoi mentioned this issue Jan 27, 2021

Allow for "emergency" push contingency during the code freeze period before the release ESMValGroup/ESMValCore#963

Closed

bjlittle mentioned this issue Jan 27, 2021

Promote unknown units to dimensionless in aux factories #3965

Merged

trexfeathers closed this as completed Jan 27, 2021

trexfeathers removed the New: Issue label Jan 27, 2021

bjlittle removed the Status: Work in Progress label Jan 27, 2021

Derived coordinates are not loaded anymore? #3961

Derived coordinates are not loaded anymore? #3961

Comments

valeriupredoi commented Jan 26, 2021

📰 Custom Issue

bjlittle commented Jan 26, 2021

bjlittle commented Jan 27, 2021 • edited Loading

bjlittle commented Jan 27, 2021 • edited Loading

bjlittle commented Jan 27, 2021 • edited Loading

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021 • edited Loading

valeriupredoi commented Jan 27, 2021

bouweandela commented Jan 27, 2021

schlunma commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

schlunma commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021 • edited Loading

bjlittle commented Jan 27, 2021 • edited Loading

valeriupredoi commented Jan 27, 2021

schlunma commented Jan 27, 2021

bjlittle commented Jan 27, 2021 • edited Loading

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021 • edited Loading

schlunma commented Jan 27, 2021 • edited Loading

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

valeriupredoi commented Jan 27, 2021

schlunma commented Jan 27, 2021

trexfeathers commented Jan 27, 2021

bjlittle commented Jan 27, 2021 • edited Loading

valeriupredoi commented Jan 27, 2021

bouweandela commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading

schlunma commented Jan 27, 2021 •

edited

Loading

bjlittle commented Jan 27, 2021 •

edited

Loading