Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have Travis test with iris-grib, remove problem tests #3469

Merged
merged 6 commits into from
Oct 22, 2019
Merged

Have Travis test with iris-grib, remove problem tests #3469

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9bc3735
Have Travis test with iris-grib, remove problem tests
stephenworsley Oct 16, 2019
d0625d4
mock GribInternalError correctly
stephenworsley Oct 16, 2019
4149e53
Update license headers
stephenworsley Oct 16, 2019
327b296
account for changes in handling of grib message defaults
stephenworsley Oct 17, 2019
95972c5
Test against the latest version of python-eccodes
stephenworsley Oct 18, 2019
11826d2
Moved irir-grib skip to iris.tests
stephenworsley Oct 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,10 @@ install:
- python setup.py --quiet install

# TODO : remove when iris doesn't do an integration test requiring iris-grib.
# TODO: uncomment and address the 5 failures and 10 errors in iris-grib.
# - if [[ "${TEST_MINIMAL}" != true ]]; then
# conda install --quiet -n ${ENV_NAME} python-eccodes;
# conda install --quiet -n ${ENV_NAME} --no-deps iris-grib;
# fi
- if [[ "${TEST_MINIMAL}" != true ]]; then
conda install --quiet -n ${ENV_NAME} python-eccodes;
conda install --quiet -n ${ENV_NAME} --no-deps iris-grib;
fi

script:
# Capture install-dir: As a test command must be last for get Travis to check
Expand Down
5 changes: 4 additions & 1 deletion lib/iris/tests/integration/test_grib2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# (C) British Crown Copyright 2014 - 2017, Met Office
# (C) British Crown Copyright 2014 - 2019, Met Office
#
# This file is part of Iris.
#
Expand All @@ -22,6 +22,7 @@
# Import iris.tests first so that some things can be initialised before
# importing anything else.
import iris.tests as tests
from unittest import skip

from cf_units import Unit
import numpy.ma as ma
Expand Down Expand Up @@ -49,6 +50,7 @@ def test_gdt1(self):
cube = load_cube(path)
self.assertCMLApproxData(cube)

@skip('iris-grib is currently causing errors')
Copy link
Contributor Author

@stephenworsley stephenworsley Oct 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is related to (and possibly solved by) this pull request.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested this with the latest version of iris-grib and I can confirm that the error does not occur with the current iris-grib master branch.

def test_gdt90_with_bitmap(self):
path = tests.get_data_path(('GRIB', 'umukv', 'ukv_chan9.grib2'))
cube = load_cube(path)
Expand Down Expand Up @@ -280,6 +282,7 @@ def test_regular(self):
cube = load_cube(path)
self.assertCMLApproxData(cube)

@skip('iris-grib is currently causing failures')
def test_reduced(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This failure seems to have been introduced by the latest verion of iris-grib. The points of one of the dimensions is reading [0.] rather than [~2.] .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When loading this file I got the warning:

UserWarning: Unable to create instance of HybridPressureFactory.
The source data contains no field(s) for 'ref_surface_pressure'.

Possibly related to this commit: SciTools/iris-grib@9f26e7a

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually related to this issue: SciTools/iris-grib#150
When the suggested changes are applied this test passes again.

path = tests.get_data_path(('GRIB', 'reduced', 'reduced_gg.grib2'))
cube = load_cube(path)
Expand Down
5 changes: 4 additions & 1 deletion lib/iris/tests/integration/test_grib_load.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# (C) British Crown Copyright 2010 - 2017, Met Office
# (C) British Crown Copyright 2010 - 2019, Met Office
#
# This file is part of Iris.
#
Expand Down Expand Up @@ -31,6 +31,7 @@
# Import iris tests first so that some things can be initialised before
# importing anything else
import iris.tests as tests
from unittest import skip

import iris
import iris.exceptions
Expand Down Expand Up @@ -64,6 +65,7 @@ def test_load_time_processed(self):
"time_bound.grib2")))
self.assertCML(cubes, ("grib_load", "time_bound_grib2.cml"))

@skip('iris-grib is currently causing errors')
def test_load_3_layer(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've done a bit of experimenting with this test. The error seems to be thrown when trying to realise one of the cubes in cubes. We get the error:
ValueError: cannot reshape array of size 103680 into shape (360,600)
Its worth noting that while cubes[1] has shape (360, 600) (where 360*600 = 12960), cubes[2] has shape (360,288) (where 360*288 = 103680).

It suggests that perhaps the cubes are at some point interfering with each other.
I tried changing the order of the cubelist to
cubes = iris.cube.CubeList([cubes[2], cubes[0], cubes[1]])
and I got the error:
RuntimeError: Invalid GRIB message: /home/travis/build/stephenworsley/iris/iris-test-data/test_data/GRIB/3_layer_viz/3_layer.grib2 @ 671744
Traceback can be found here.
I'm still not sure if this is an Iris or Iris-grib issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, if the CubeList isn't reordered then the test simply fails without errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I reordered the Cubelist as:
cubes = iris.cube.CubeList([cubes[0], cubes[2], cubes[1]])
I got the error:
gribapi.errors.PrematureEndOfFileError: End of resource reached when reading message

The traceback can be found here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that if all the data is realised first, the test succeeds. The data must, however, be realised in the correct order:

cubes[0].data
cubes[1].data
cubes[2].data

where the data in cubes is realised the step before it is reordered.
If cubes[1].data is realised first, it will cause an error (similarly for cubes[2]).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This problem seems to be related to the package python-eccodes. I can replicate this problem when I create a python 3 environment with conda install python-eccodes but not with conda install eccodes. I have had dificulty trying to have travis install eccodes though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for some reason, the most recent version of python-ecodes is 0.9.3 rather than 2.12.3 . When the travis file specifies python-eccodes=0.9.3 then this test no longer errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, this issue is discussed here: conda-forge/python-eccodes-feedstock#49
It looks like in future, a python interface will be added to eccodes so that it will replace python-eccodes and solve this version problem: conda-forge/staged-recipes#9731

cubes = iris.load(tests.get_data_path(('GRIB', "3_layer_viz",
"3_layer.grib2")))
Expand Down Expand Up @@ -131,6 +133,7 @@ def test_reduced_ll(self):
("GRIB", "reduced", "reduced_ll.grib1")))
self.assertCML(cube, ("grib_load", "reduced_ll_grib1.cml"))

@skip('iris-grib is currently causing failures')
def test_reduced_gg(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This failure seems to have been introduced by the latest verion of iris-grib. The points of one of the dimensions is reading [0.] rather than [~2.] .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed by the changes suggested in SciTools/iris-grib#150

cube = iris.load_cube(tests.get_data_path(
("GRIB", "reduced", "reduced_gg.grib2")))
Expand Down
6 changes: 3 additions & 3 deletions lib/iris/tests/test_grib_load_translations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def _mock_gribapi_fetch(message, key):
if key in message:
return message[key]
else:
raise _mock_gribapi.GribInternalError
raise _mock_gribapi.errors.GribInternalError
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gribapi has changed where GribInternalError is stored. Fixing this solves 8 of the 10 errors in the grib tests mentioned in #3447. I've decided not to handle the possibility of both cases since the updated version of iris-grib is the only one which is compatible with python3.



def _mock_gribapi__grib_is_missing(grib_message, keyname):
Expand All @@ -83,13 +83,13 @@ def _mock_gribapi__grib_get_native_type(grib_message, keyname):
"""
if keyname in grib_message:
return type(grib_message[keyname])
raise _mock_gribapi.GribInternalError(keyname)
raise _mock_gribapi.errors.GribInternalError(keyname)


if tests.GRIB_AVAILABLE:
# Construct a mock object to mimic the gribapi for GribWrapper testing.
_mock_gribapi = mock.Mock(spec=gribapi)
_mock_gribapi.GribInternalError = Exception
_mock_gribapi.errors.GribInternalError = Exception

_mock_gribapi.grib_get_long = mock.Mock(side_effect=_mock_gribapi_fetch)
_mock_gribapi.grib_get_string = mock.Mock(side_effect=_mock_gribapi_fetch)
Expand Down
6 changes: 5 additions & 1 deletion lib/iris/tests/test_grib_save.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# (C) British Crown Copyright 2010 - 2017, Met Office
# (C) British Crown Copyright 2010 - 2019, Met Office
#
# This file is part of Iris.
#
Expand All @@ -20,6 +20,7 @@

# import iris tests first so that some things can be initialised before importing anything else
import iris.tests as tests
from unittest import skip

import os
import datetime
Expand All @@ -44,6 +45,7 @@ class TestLoadSave(tests.TestGribMessage):
def setUp(self):
self.skip_keys = []

@skip('iris-grib is currently causing failures')
def test_latlon_forecast_plev(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the following three tests are failing due to a change introduced by this commit. It may be appropriate to change the expected_diffs.

source_grib = tests.get_data_path(("GRIB", "uk_t", "uk_t.grib2"))
cubes = iris.load(source_grib)
Expand All @@ -63,6 +65,7 @@ def test_latlon_forecast_plev(self):
expect_diffs, self.skip_keys,
skip_sections=[2])

@skip('iris-grib is currently causing failures')
def test_rotated_latlon(self):
source_grib = tests.get_data_path(("GRIB", "rotated_nae_t",
"sensible_pole.grib2"))
Expand All @@ -85,6 +88,7 @@ def test_rotated_latlon(self):
expect_diffs, self.skip_keys,
skip_sections=[2])

@skip('iris-grib is currently causing failures')
def test_time_mean(self):
# This test for time-mean fields also tests negative forecast time.
source_grib = tests.get_data_path(("GRIB", "time_processed",
Expand Down