Detailed report for testing.assert_equal and testing.assert_identical #1507

benbovy · 2017-08-11T09:38:23Z

~~Closes #xxxx~~
Tests added / passed
Passes git diff upstream/master | flake8 --diff
Fully documented, including whats-new.rst for all changes and api.rst for new API

~~In addition to Dataset repr, the error message also shows the output of Dataset.info() for both datasets.~~

~~This may not be the most elegant solution, but it is helpful when datasets only differ by their attributes attached to coordinates or data variables (not shown in repr). I'm open to any suggestion.~~

The report shows the differences for dimensions, data values (Variable and DataArray), coordinates, data variables and attributes (the latter only for testing.assert_identical).

There is currently not much tests for xarray.testing functions, but I'm willing to add more if needed.

Not sure if it's worth a what's new entry (EDIT: added one).

jhamman

I'm fine adding something like this. Another option would be to add a verbose repr that prints value, attributes, and encoding. This could be tacked on to .info() or a new method. Currently, this is designed to look like the output from the ncdump command line utility.

jhamman · 2017-08-11T18:47:36Z

xarray/testing.py

+        assert a.identical(b), (
+            '{}\n{}\n---\n{}\n{}'
+            .format(a, b, _get_info_as_str(a), _get_info_as_str(b))
+        )


if this is a xr.Variable instance, this will not work since it won't have the info method.

shoyer · 2017-08-11T21:41:42Z

This is certainly welcome. In an ideal world, we might point specifically to the exact differing variables/attributes, but this is a fine start.

benbovy · 2017-08-11T22:36:38Z

Indeed, exact differing variables/attributes would be much better here. I'll look into this next week.

pytest's failure reports and reports of numpy.testing functions may be good sources of inspiration.

max-sixty · 2019-01-14T21:47:16Z

Any interest in pushing this over the line @benbovy ?

benbovy · 2019-01-15T18:19:44Z

Thanks for reminding me this PR @max-sixty ! It deserves more than no commits since 1.5 years!

The assertion error messages now look like pytest's failure reports (see test_eq_dict). I've tried to reuse as much as possible the repr formatting that we already have for the xarray objects.

As examples:

ds_a = xr.Dataset(data_vars={'var1': (('x', 'y'), [[1, 2, 3], [4, 5, 7]]),
                             'var2': ('x', [3, 4])},
                  coords={'x': ['a', 'b'], 'y': [1, 2, 3]},
                  attrs={'units': 'm', 'description': 'desc'})

ds_b = xr.Dataset(data_vars={'var1': ('x', [1, 2])},
                  coords={'x': ('x', ['a', 'c'], {'source': 0}), 'label': ('x', [1, 2])},
                  attrs={'units': 'kg'})

>>> xr.testing.assert_identical(ds_a, ds_b)
AssertionError: Left and right Dataset objects are not identical
Differing dimensions:
(x: 2, y: 3) != (x: 2)
Differing coordinates:
L * x        (x) <U1 'a' 'b'
R * x        (x) <U1 'a' 'c'
    source: 0
Left contains more coordinates:
  * y        (y) int64 1 2 3
Right contains more coordinates:
    label    (x) int64 1 2
Differing data variables:
L   var1     (x, y) int64 1 2 3 4 5 7
R   var1     (x) int64 1 2
Left contains more data variables:
    var2     (x) int64 3 4
Differing attributes:
L   units: m
R   units: kg
Left contains more attributes:
    description: desc

da_a = xr.DataArray([[1, 2, 3], [4, 5, 7]],
                    dims=('x', 'y'),
                    coords={'x': ['a', 'b'], 'y': [1, 2, 3]},
                    attrs={'units': 'm', 'description': 'desc'})

da_b = xr.DataArray([1, 2],
                    dims='x',
                    coords={'x': ['a', 'c'], 'label': ('x', [1, 2])},
                    attrs={'units': 'kg'})

>>> xr.testing.assert_equal(da_a, da_b)
AssertionError: Left and right DataArray objects are not equal
Differing values:
L
    array([[1, 2, 3],
           [4, 5, 7]])
R
    array([1, 2])
Differing coordinates:
L * x        (x) <U1 'a' 'b'
R * x        (x) <U1 'a' 'c'
Left contains more coordinates:
  * y        (y) int64 1 2 3
Right contains more coordinates:
    label    (x) int64 1 2

Those examples are probably not the most readable ones but it's for a full showcase.

A downside of this approach (i.e., report the exact differing) is that when a and b differs, full comparison is done once again for formatting the report. I don't think there is an easy way to avoid that, but I don't think it's a big deal either.

xarray/core/formatting.py

max-sixty · 2019-01-15T18:36:12Z

full comparison is done once again for formatting the report. I don't think there is an easy way to avoid that, but I don't think it's a big deal either.

i.e. there's a performance cost? Yeah that's def fine!

max-sixty · 2019-01-15T18:37:02Z

This looks amazing! I'm excited to use it. Thanks for finishing it up

max-sixty · 2019-01-15T18:39:53Z

If you want to add that example as a test, that could be a good way of documenting the function. But I don't think it's strictly needed

benbovy · 2019-01-15T18:43:12Z

I'm wondering which is the best among the options below:

A.

Differing data variables:
L   var1     (x, y) int64 1 2 3 4 5 7
R   var1     (x) int64 1 2
L   var2     (x, y) int64 4 5 6 7 8 9
    description: variable 2 
R   var2     (x, y) int64 4 5 6 7 8 9

B.

Differing data variables:
L   
    var1     (x, y) int64 1 2 3 4 5 7
R
    var1     (x) int64 1 2
L
    var2     (x, y) int64 4 5 6 7 8 9
    description: variable 2 
R
    var2     (x, y) int64 4 5 6 7 8 9

C.

Differing data variables:
L   
    var1     (x, y) int64 1 2 3 4 5 7
    var2     (x, y) int64 4 5 6 7 8 9
    description: variable 2
R
    var1     (x) int64 1 2
    var2     (x, y) int64 4 5 6 7 8 9

benbovy · 2019-01-15T18:46:36Z

If you want to add that example as a test, that could be a good way of documenting the function. But I don't think it's strictly needed

Yes I could use those examples for the tests. I think it might be good to write shallow tests at least for diff_array_repr and diff_dataset_repr (in formatting.py).

max-sixty · 2019-01-15T19:14:42Z

I think you should feel free to merge an MVP rather than perfecting, but check out https://docs.pytest.org/en/latest/example/simple.html#writing-well-integrated-assertion-helpers

I think that might be as simple as wrapping these in pytest.fail(...)

max-sixty · 2019-01-15T19:19:53Z

I'm wondering which is the best among the options below:

TBH I think they're all good; I don't have a strong view. I'd low-confidence rank A->C->B

dcherian · 2019-01-15T19:58:02Z

I agree with @max-sixty A > C > B but only weakly.

pep8speaks · 2019-01-16T11:07:35Z

Hello @benbovy! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on January 16, 2019 at 13:27 Hours UTC

max-sixty · 2019-01-16T11:52:45Z

This looks v good @benbovy!

benbovy · 2019-01-16T11:56:04Z

I agree with @max-sixty A > C > B but only weakly.

Same opinion.

This looks v good @benbovy!

I'm just fixing int32 vs int64 issue with tests run on appveyor, then to me it's ready to merge unless someone else has objections.

shoyer

This looks fantastic!

shoyer · 2019-01-16T12:49:26Z

xarray/tests/test_formatting.py

+        Differing coordinates:
+        L * x        (x) <U1 'a' 'b'
+        R * x        (x) <U1 'a' 'c'
+        Left contains more coordinates:


"Left contains more coordinates" sounds a little funny to me.

Maybe "Coordinates on the left DataArray but not the right" or "Coordinates only on the left object"?

Agreed! I followed pytest's reports too blindly here.

benbovy · 2019-01-18T08:38:01Z

If everyone is happy with this I'm going to merge it.

shoyer · 2019-01-18T09:09:46Z

If everyone is happy with this I'm going to merge it.

Yes, please!

max-sixty · 2019-01-18T14:52:09Z

Thanks @benbovy!

* master: stale requires a label (pydata#2701) Update indexing.rst (pydata#2700) add line break to message posted (pydata#2698) Config for closing stale issues (pydata#2684) to_dict without data (pydata#2659) Update asv.conf.json (pydata#2693) try no rasterio in py36 env (pydata#2691) Detailed report for testing.assert_equal and testing.assert_identical (pydata#1507) Hotfix for pydata#2662 (pydata#2678) Update README.rst (pydata#2682) Fix test failures with numpy=1.16 (pydata#2675)

* refactor-plot-utils: (22 commits) review comment. small rename stale requires a label (pydata#2701) Update indexing.rst (pydata#2700) add line break to message posted (pydata#2698) Config for closing stale issues (pydata#2684) to_dict without data (pydata#2659) Update asv.conf.json (pydata#2693) try no rasterio in py36 env (pydata#2691) Detailed report for testing.assert_equal and testing.assert_identical (pydata#1507) Hotfix for pydata#2662 (pydata#2678) Update README.rst (pydata#2682) Fix test failures with numpy=1.16 (pydata#2675) lint Back to map_dataarray_line Refactor out cmap_params, cbar_kwargs processing Refactor out colorbar making to plot.utils._add_colorbar flake8 facetgrid refactor Refactor out utility functions. ...

jhamman requested changes Aug 11, 2017

View reviewed changes

jhamman added the topic-testing label Aug 11, 2017

more detailed AssertionError message for assert_identical

53f80b3

benbovy force-pushed the assert_identical_msg branch from 4298eba to 53f80b3 Compare January 15, 2019 08:16

print differing dimensions/data/variables/attributes

9edfb9b

benbovy commented Jan 15, 2019

View reviewed changes

xarray/core/formatting.py Show resolved Hide resolved

benbovy added 4 commits January 16, 2019 09:41

minor tweaks

07868e2

Merge remote-tracking branch 'origin/master' into assert_identical_msg

1578019

add what's new entry

5f4f87b

add tests for diff_array_repr and diff_dataset_repr

1469cde

pep8

f670aa3

benbovy added 2 commits January 16, 2019 13:46

add differing dimensions in diff_array_repr

6f0f704

fix tests (explicit numpy dtypes)

a4721ac

shoyer approved these changes Jan 16, 2019

View reviewed changes

fix tests (dtype shown / not shown in array repr)

b6f4faa

minor tweaks

443e593

benbovy changed the title ~~More detailed AssertionError message for assert_identical~~ Detailed report for testing.assert_equal and testing.assert_identical Jan 16, 2019

benbovy merged commit 1d0a2bc into pydata:master Jan 18, 2019

benbovy mentioned this pull request Jan 18, 2019

Add names for test failures #1690

Closed

4 tasks

benbovy deleted the assert_identical_msg branch October 25, 2019 15:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detailed report for testing.assert_equal and testing.assert_identical #1507

Detailed report for testing.assert_equal and testing.assert_identical #1507

benbovy commented Aug 11, 2017 •

edited

Loading

jhamman left a comment •

edited

Loading

jhamman Aug 11, 2017

shoyer commented Aug 11, 2017

benbovy commented Aug 11, 2017

max-sixty commented Jan 14, 2019

benbovy commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

benbovy commented Jan 15, 2019

benbovy commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

dcherian commented Jan 15, 2019

pep8speaks commented Jan 16, 2019 •

edited

Loading

max-sixty commented Jan 16, 2019

benbovy commented Jan 16, 2019

shoyer left a comment

shoyer Jan 16, 2019

benbovy Jan 16, 2019

benbovy commented Jan 18, 2019

shoyer commented Jan 18, 2019

max-sixty commented Jan 18, 2019

Detailed report for testing.assert_equal and testing.assert_identical #1507

Detailed report for testing.assert_equal and testing.assert_identical #1507

Conversation

benbovy commented Aug 11, 2017 • edited Loading

jhamman left a comment • edited Loading

Choose a reason for hiding this comment

jhamman Aug 11, 2017

Choose a reason for hiding this comment

shoyer commented Aug 11, 2017

benbovy commented Aug 11, 2017

max-sixty commented Jan 14, 2019

benbovy commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

benbovy commented Jan 15, 2019

benbovy commented Jan 15, 2019

max-sixty commented Jan 15, 2019

max-sixty commented Jan 15, 2019

dcherian commented Jan 15, 2019

pep8speaks commented Jan 16, 2019 • edited Loading

Comment last updated on January 16, 2019 at 13:27 Hours UTC

max-sixty commented Jan 16, 2019

benbovy commented Jan 16, 2019

shoyer left a comment

Choose a reason for hiding this comment

shoyer Jan 16, 2019

Choose a reason for hiding this comment

benbovy Jan 16, 2019

Choose a reason for hiding this comment

benbovy commented Jan 18, 2019

shoyer commented Jan 18, 2019

max-sixty commented Jan 18, 2019

benbovy commented Aug 11, 2017 •

edited

Loading

jhamman left a comment •

edited

Loading

pep8speaks commented Jan 16, 2019 •

edited

Loading