-
Notifications
You must be signed in to change notification settings - Fork 285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix imagehash hex_to_hash #2989
Conversation
The graphic test
This change doesn't concern me and we don't know how long it's been failing for, so i suggest we just accept the difference. |
Also see the associated PR SciTools/test-iris-imagehash#13 that adds the new image to the Until the |
Created issue #2991 to cover the last failing test to investigate, which is not |
@bjlittle You need to spend more time sleeping and less time working. |
Clearly this PR fixes the image tests. However, on reflection seeing the idiff changes, it's not pretty handling one imagerepo with mixed old (faulty) and new hashes. |
@bjlittle Will the test-iris-imagehash repo just keep getting bigger really fast, or is this new image hash generation just a one-off? I suppose that's something that we just don't know, but if we get an issue like this again then we could just keep adding to the imagehash repo(s), right? |
Also, would we after a little while remove the old faulty image hashes and just use the new ones, or would we have to use both sets forever? |
@corinnebosley Going forwards, we'll definitely need to retain the old hashes, so that prior versions of iris will have functional image testing. If we remove the old hashes in That said, it should be possible to not double up the size of the With this approach, we keep the old and new separate, we don't double up the This seems like a cleaner approach to me, plus we can (easily) rip the perceptual imagehash functionality out of iris into a separate package, if we want to so that at a future date. The What do you think? Does this make sense? |
Okay, just went ahead and made the necessary changes to the The above PR was merged into my master https://github.com/bjlittle/imagehash To test this PR, commit ea02532 has it's This approach has has worked out quite nicely and I much prefer it as a way forward. The code changes in iris are minimal whilst still allowing us the freedom to make graphical perceptual image hashing testinggeneric at a later date. |
The latest commit 5e5d262 changes the So this PR requires SciTools/test-iris-imagehash#14 to be merged first to work! |
Analysis of the old hash to new hash cross-referencing highlighted that there are duplicate entries in the
The two hashes that have duplicates but tests with different actual result images are okay - what this means is that the differences are "similar enough" as represented within the 16byte perceptual image hash and the (expected - actual) hash difference hamming distance of only 2bits. If we wanted to discriminate between these images then we should either:
I recommend neither of these options, as it will increase the number of items in the |
@pelson We could really do with getting this unblocked ... any chance of a review? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
If we ever want to release a 2.0.1, then I suggest we need to change the target branch.
In truth though, I'm happy to move forward to a 2.1 in short order.
if action == 'first': | ||
index = 0 | ||
elif action == 'last': | ||
index = -1 | ||
elif action == 'similar': | ||
index = np.argmin(distances) | ||
elif action == 'difference': | ||
elif action == 'different': |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😕. What's happening here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was always wrong! See https://github.com/SciTools/iris/blob/master/lib/iris/tests/idiff.py#L288
@@ -1,725 +1,726 @@ | |||
{ | |||
"example_tests.test_COP_1d_plot.TestCOP1DPlot.test_COP_1d_plot.0": [ | |||
"https://scitools.github.io/test-iris-imagehash/images/5dff1a996c06b47193eecc52275b936d58b6239ba94c133643897c6c2333f086.png" | |||
], | |||
"https://scitools.github.io/test-iris-imagehash/images/v4/baff589936602d8ec977334ae4dac9b61a6dc4d99532c86cc2913e36c4cc0f61.png" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit awkward, because technically I should double check that the old hash matches the new (and that a change hasn't been introduced).
I'm not going to do that, and rely on the fact that the tests passed before this change, and that they will pass after this change + the fact that no changes have occurred that change the behaviour of Iris' plotting...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's really pretty difficult for a reviewer to review the hash changes, but I've tried to address that some what with this #2989 (comment)
Bill I re-triggered the build, but got:
Anything I can do to help? |
@pelson Fixed 🎉 I just needed to fix the last test to point to the new This LGTM, but to be honest it should really also be merged into Now that the image tests are working, that would be a good win, and a refreshing change to turnaround a quick release on the heels of |
@corinnebosley @pp-mo @dkillick @pelson Please someone merge this PR... before I have to change the license headers to say |
* Fix coord_comparison when coords are the same object * Added what's new items for SciTools#2730 and SciTools#2325. (SciTools#2852) * Added what's new items for SciTools#2730 and SciTools#2325. [skip ci] * Moved the bugfix docs to enhancement for the 3x speedup of area weights. [skip ci] * Add unit test * Add workaround * Add test of current dask behaviour * Ammended cube.var_name documentation. (SciTools#2874) Ammended cube.var_name documentation to refer to it as NetCDF. * Futures netcdf no unlimited (SciTools#2838) * switched boolean operator and altered docstring * raised error on future flag and tweaked functionality * small change to deprecation warning message * Code changes * remake cdl results * More CDL updates (cherry picked from commit 47272b3) * Review actions * Use default Dask scheduler settings (SciTools#2879) Disable dask.set_options call on _lazy_data import. * Fix doctest failing due to change in 'netcdf_promote' future behaviour (SciTools#2883) * Remove unneeded image repo url (SciTools#2885) * Updated 2.0 release notes. (SciTools#2884) * Add iris-grib as a testing dependency; Python 3 included. * Fix requirements for iris_grib>=0.12, assuming it will have eccodes + thus work with Python 3 * Made the iris extensions not install by default. * Reduced the number of processes used by default when testing. * Set the documentation build to use viridis as the default colormap (rather than Jet as we are using matplotlib 1 still). * Change behaviour of 'error' level (SciTools#2881) * Change behaviour of 'error' level * Change Future netcdf_no_unlimited tests * Fix tests testing the wrong thing * Dask processing options docs updates * Added some what's new information about SciTools#2435 and SciTools#2810. * Use the cube dimension order to determine the plot axes. * Small update to experimental-FF whatsnew statement (SciTools#2898) * Move some whatsnew items, remove spurious list * Move nc-no-unlim text to whatsnew * Fill value handling on save notes * Netcdf scalar cube loading bugfix Previously Iris used to simply ignore scalar cubes. * Corrected usage context in description of PointInCell regrid scheme. (SciTools#2735) * Re-worked setup.py to avoid the need for separate/non-standard build commands (SciTools#2891) Re-worked setup.py to avoid the need for separate/non-standard build commands. * Missing data whitepaper (SciTools#2906) * Ensure lon/lat return order is consistent throughout (SciTools#2238) * Updated to v46 of the CF standard-name table. (SciTools#2908) * Updated to v46 of the CF standard-name table. * Updated the area-type table to v6. * Use viridis cmap for doc images (SciTools#2909) * Stickler config (SciTools#2896) * Added a flake8 config. * Added a stickler-ci config. * Handled the recent update to conda-forge for testing, perhaps relating to GEOS update. (SciTools#2912) * Bumped master to the v2.1 stream. (SciTools#2910) * Stickler config (SciTools#2896) * Added a flake8 config. * Added a stickler-ci config. * Improved the way requirements are handled, ensuring that pip gets all the information it needs. (SciTools#2890) Improved the way requirements are handled, ensuring that pip gets all the information it needs. * Further docs refinement for v2.0.0 (SciTools#2916) Some minor documentation tweaks for v2.0.0 * Remove obsolete reference from Userguide. (SciTools#2915) * Reverse 2878 (SciTools#2926) * Remove dask issue workaround; wrap computed MaskedConstant as array * Reduce exception message for loading missing files. (SciTools#2925) * reduce error message * update file not found tests * clearer message * Allow missing value (SciTools#2892) * Allow missing_value as a user set attribute This allows the user to set missing_value as an attribute on a cube while ensuring that it is saved as local attribute in the netcdf file. * Added the what's new and updated the copyright years for SciTools#2892. * Fix some header dates. * Add note on using intersection (SciTools#2931) * CML changes for cf_units v1.2.0 (SciTools#2936) * CML changes for cf-units v1.2.0 * Add whatsnew entry * [DOC]: Update userguide note on intersection (SciTools#2933) * Update userguide note on intersection * Update with concrete boundary examples * Prepare master for v2.0.x branch (SciTools#2934) * Converted the contributions into the whatsnew. * Updated to a dev version string. * Tweaked what's new, bumped version string, and updated the MANIFEST for pypi. * Perserve dtype of source cube with area weighted regridder (SciTools#2203) Perserve dtype of source cube with area weighted regridder & add what's new * Update license headers to handle the merge of a PR from 2017 in 2018 (SciTools#2948) * DOC: lazy data docstring made clearer (SciTools#2947) * DOC: lazy data docstring made clearer * Clarify "view" * Remove gdal from 'all' to 'extensions' depedency group. * Remove gdal from 'all' to 'extensions' depedency group. * DOC: lazy data docstring made clearer (SciTools#2947) * DOC: lazy data docstring made clearer * Clarify "view" * No docs are better than wrong docs on dask * Add a general zenodo link to the readme. (SciTools#2955) * Improved the pip installability of scitools-iris. (SciTools#2956) * Release of iris v2.0.0 🎉 * Merge v2.0.x into master (SciTools#2957) * Tweaked what's new, bumped version string, and updated the MANIFEST for pypi. * Remove gdal from 'all' to 'extensions' depedency group. * DOC: lazy data docstring made clearer (SciTools#2947) * DOC: lazy data docstring made clearer * Clarify "view" * No docs are better than wrong docs on dask * Add a general zenodo link to the readme. (SciTools#2955) * Improved the pip installability of scitools-iris. (SciTools#2956) * Release of iris v2.0.0 🎉 * Move v2.0.x branch into development mode. * Fix the documentation building for latest version of sphinx. (SciTools#2958) * Remove FutureWarning from use of np.issubdtype(obj, np.str) (SciTools#2954) * Add cube co-realisation. (SciTools#2967) * Fix for changed API of cartopy.crs.Stereographic (SciTools#2971) * Fix for changed API of cartopy.crs.Stereographic; fixes tests.test_coordsystem. * License header fix. * remove site.cfg exclude from manifest.in (SciTools#2978) * Save calendar attribute for cube data (SciTools#2984) * Fix bug with aux coords with scalefactor/add_offset (SciTools#2975) Fix bug with aux coords with scalefactor/add_offset * Rename to correct package name * Fix imagehash hex_to_hash (SciTools#2989) * Fix imagehash hex_to_hash * expose hamming distance vector * fix idiff * update imagerepo * fix repo hashes (bjlittle) * fix repo hashed (scitools) * fix test_image_json * test_pearsonr: use compatible cubes for testing compatible cubes (SciTools#2960) Pearsonr test using compatible cubes * Use unittest.mock for Python 3 (SciTools#3003) * reflect new latitude guess_bounds behaviour (SciTools#3006) * Pin Dask to >=0.17.1 (SciTools#2965) * Fill a usability gap in iris.analysis.Trajectory (SciTools#2770) * Fill a usability hole in Trajectory * Remove iris.experimental.um module (SciTools#2781) * Remove iris.experimental.um module * Replace use of sphinx's status_iterator function, which has moved * Lazy unit conversions for cube and coord data. (SciTools#2964) * Gracefully handle long time intervals (SciTools#2354) Gracefully handle long time intervals (months & years) when plotting * Prevent creation of invalid CF variable names. (SciTools#3009) Prevent creation of invalid CF variable names. * Add stash check to name() (SciTools#2930) Try the STASH attribute when checking for possible names (as is already done in the summary method) * Attempts to fix docs-build error reporting. (SciTools#2983) Remove final semicolon to report docs-build error. * Phenomenon translation updates (SciTools#3011) * Netcdftime dep (SciTools#3017) Pin to netcdf4 1.3 or below while we await netdftime updates Correct deprecated pandas use in v0.23 * add 'axes' kwarg to _label when called from contourf * Replace iris.co_realise_cubes with CubeList.realise_data. (SciTools#3013) * Replace iris.co_realise_cubes with CubeList.realise_data. * Avoid exposing lazy-data functions in iris.cube and iris.coord modules. * Tidy lazy_data imports in cube + coords modules. * Amend var_name documentation (SciTools#3020) Amend documentation to be explicit that var_name is a netCDF quantity, not necessarily a CF one. * Updated tests for cf_units 2 (SciTools#3023) Updated tests for cf_units>=2 This change: * Removes the redundant ``iris/tests/test_unit.py`` tests. They are all migrated to cf_units. * Makes use of CALENDAR_GREGORIAN, rather than CALENDAR_STANDARD. These calendars are implemented as aliases in cf_units 2, and the "STANDARD" concept is not preserved/round-tripped. Added a what's new to help reduce possible impact for users. * Use ConfigParser for py>=3.2 (SciTools#3025) * WIP: Unpin mpl (SciTools#3019) * unpin mpl * Add _repr_html_ for Iris cubes (SciTools#2918) Add cube _repr_html_ functionality * Partial collapse of multi-dimensional coordinates (SciTools#3028) Adding support for partial collapse of multi-dimensional coordinates * Tidy up the root of the repository for v2.1 (SciTools#3034) A bunch of miscellaneous changes in preparation for the v2.1 release: * Move CONTRIBUTING.md to a .github/ folder. See also: https://help.github.com/articles/setting-guidelines-for-repository-contributors/ * Updated the .gitignore to reflect what gets produce in day-to-day iris development * Reduced the amount of redundancy in INSTALL. It now points to the requirements/ directory which is what we actually test. * Updated minimum dependencies in requirements (because that is all we test): * Numpy>=1.14 * Matplotlib>=2 * cf_units>=2 * Remove the CODING matrix item. (SciTools#3036) * Add Albers Equal Area Projection and some test as per SciTools#2496 (SciTools#2943) Add Albers Equal Area Projection and NetCDF rules to handle it * Improve the appeal of the landing pages (docs and README) (SciTools#3040) * Update the Iris README markdown to be consistent with other SciTools projects. * Gave the package a description (for pypi). * Minor spruce of the Iris docs layout to include the tagline and the forkme button * Use the correct pypi name for nc-time-axis. (SciTools#3039) * Use cftime instead of netcdftime (SciTools#3016) * use cftime instead of netcdftime * fix the docs too * fix license year * review actions * simplify test_pandas * what's new * remove cf_units check * Fixed the intermittent test in SciTools#3035. (SciTools#3037) * Fixed the intermittent test in SciTools#3035. The problem was that iris.util.format_array was stateful (via np.set_printoptions). * Fix stateful tests (SciTools#26) * Add dask-array as a requirement when pip installing. (SciTools#3033) * Unpin netcdf4 (SciTools#3031) * unpin netcdf4 * adapt for netcdf4 >=1.4 * fix userguide doctests * Metarelate updates in the UM 2 CF mapping (SciTools#3043) * Updated the README, and copied that over to the iris docs homepage. (SciTools#3045) * Added the what's new for iris 2.1 (SciTools#3044) Added the what's new for iris 2.1 * Updated the cf-standard names to v54. (SciTools#3046) Achieved with: $ wget http://cfconventions.org/Data/cf-standard-names/54/src/cf-standard-name-table.xml -O etc/cf-standard-name-table.xml ... --2018-06-02 06:32:17-- http://cfconventions.org/Data/cf-standard-names/54/src/cf-standard-name-table.xml ... * iris.quickplot: Ensure labels and plot use the same axes Allow iris.quickplot to accept an axes kwarg to which title and axes labels can be attached. This is done without using the matplotlib state machine. Closes SciTools#2946 * sticklering * Full-featured Mercator coord system (SciTools#3041) Add ``standard_parallel`` to the Mercator Coord System * Updated the branding and execution of Iris docs (SciTools#3050) * Updated the branding and execution of Iris This includes: * removing the "easy-to-use" statement in Iris' strapline. Let's let users be the judge of that. * improved the styling of the homepage, including re-instating the search bar. * Added a what's new for SciTools#3010. * Added a extlinks section for convenient linking to github issues with :issue:`number` and :pr:`number`. * Added a favicon for the docs. * Improve strapline style, content (SciTools#27) * Further improvements to Iris docs * Improved font (noto-sans) * Removed old CSS * Fixed a few what's new entries * [DOC] Iris Introduction updates (SciTools#3052) Update the introduction of Iris content. * Updated to v2.1.0. (SciTools#3051) * Move into development mode for v2.1.1 * Move into development mode for v2.2.0 * remove first 'and' from tagline to make it read easier * Notebook repr whatsnew update * remove first 'and' from tagline to make it read easier * minor requirements tidy * http -> https for copybutton.js (SciTools#3066) * Explain use of saver='string' in iris.save (SciTools#3062) * Fix reStructuredText warnings in documentation (SciTools#3069) * Add information about comparing STASH objects with strings (SciTools#3072) * Doc: Fallback to no syntax hightlighting without warning. (SciTools#3071) Fixes: iris/docs/iris/src/developers_guide/gitwash/development_workflow.rst:317: WARNING: Could not lex literal_block as "python". Highlighting skipped. http://www.sphinx-doc.org/en/stable/config.html#confval-highlight_language * Quick fix to get tests passing by pinning dask version (SciTools#3086) * ENH: Working zonal mean linear regridding for circular sources or with use of extrapolation (SciTools#3085) * Working zonal mean from circular/extrapolated source to target * MAINT: Refactor of zonal mean testing * MAINT: Documentation changes from review * Avoid pandas deprecation warning. (SciTools#3079) Avoid pandas deprecation warning. This also simplifies code by being explicit about usage of "base" and "values" for numpy and pandas objects, and removes legacy code that may no longer be valid. * Workaround for dask array copy bug (SciTools#3088) * add contact options to README * Add stash mapping changes to v2.1.0 changelog * Pin Dask for avoid 0.18.2 bug with masked arrays. * Tiny fix for dask, Python3 only? * Disable doctests for Python 2. * Ignore warnings and update array printouts. * Better way to disable Python2 doctests. * Support plotting 2D bounded coords * Add gridcell_angles and rotate_grid_vectors to iris.analysis.cartography, with tests: INCOMPLETE WIP * Roughly working, snapshotted with complex test plot code, to be reduced. * Small improvements. * First working quiver+streamplot. * Handle custom coords correctly * some words for docs, WIP probably * suggested tweaks and corrections * Update docs to included 2.2 versions (SciTools#3110) * Provide a test skipper for 2d coords WIP. (SciTools#3099) * Change version number (SciTools#3118) * re-added bounds definition for contiguity check, removed unnecessary test * Ensure Sphinx autodocs for grid_angles routines. * Codestyle fixes. * Whatsnew entries for 2d vector support. * Pin Dask to avoid 0.18.2 bug with masked arrays. (SciTools#3127) * Put orca_util routines in subpackage of iris.test.stock, to get existing iris.tests.unit.plot.test_2d_coords working. * Disable broken misused testcode in test_gridcell_angles. * Ditch test_gridcell_angles, none of it is functional. * Further style fixes. * Skip tests using iris-test-data, for Travis TEST_MINIMAL phases. * Codestyle fix (though this code obsolete anyway). * Fix unused imports. * Made-up test cube replaces use of iris-test-data. * Renamed keyword; improved docstring. * Change 'co' to 'coord' for clarity. * Review changes. * Update pp save rules to handle time means (SciTools#3132) * Vector plots 2 (SciTools#3120) * Small improvements; first sensible tests. * Enhanced testing; better checking and crs awareness in grid_angles routine. * Remove crud from test_gridcell_angles. * Use degree units for everything in _grid_angles. * Make assertArrayAllClose print details when it fails. * Rework and extend testing for gridcell_angles. * Fix assertArrayAllClose; remove debug code from test_gridcell_angles. * Remove obsolete assignments. * Remove obsolete code. * Small comment improvements. * Attempt to clarify docstrings of low-level routines. * More tests, and some functional fixes. * Codestyle fixes. * Review changes + fixes. * Avoid using sample data. * Temporary pin to avoid problems caused by Proj4 v5.1. (SciTools#3150) * Extending ORCA blockplotting (w/ tests) (SciTools#3135) * Tests for rotate_grid_vectors (SciTools#3148) * Tests for rotate_grid_vectors. * Small fix to assertArrayAllClose for values near 0. * Small tweaks. * Fix test method. * Fix 1 test for 'equal_nans' no longer the default. * Review changes. * Remove 'skip_2d' test decorator. (SciTools#3156) * Add integration graphics tests for pcolormesh with 2d coords (SciTools#3149) * Vector plots 4 (SciTools#3158) * Fix contig_ tolerance kwarg bug (SciTools#3162) * Add simple lazy aggregations * Lazy SUM and COUNT, tests improved * Add tests for MAX and MIN * Update tests for STD_DEV * Use NetCDF variable's chunks on load (SciTools#3131) Preserve NetCDF chunks on dataset load * Iteration and less duplication * Attempt to fix Matplotlib issue by resticting to 2.x * Non-test review actions * Make COUNT tests truer unit tests * Convert MAX and MIN tests to truer unit tests * Convert COUNT and SUM agg tests to truer unit tests * Tests for new _lazy_data function * Fix for netCDF4 1.4 behaviour change. (SciTools#3177) * Improve gracefully filling warning (SciTools#3171) * Realise that running the tests is simpler than I thought * Make Gracefully warning only appear when filling is required * Add tests of warnings * Get rid of install changes, they are in SciTools#3170 * Update after review * Add whatsnew entry * Minor clarification to entry via a little more punctuation * Revert nc vars masking fix (temporarily). (SciTools#3178) * fix for pandas column object bug (SciTools#3176) * turn columns object into pandas array, plus test included cml this time * made things the way pandas like it; bambooish * regenerated cml to match output * Update module headers and installation instructions (SciTools#3170) * Update module headers and installation instructions * Realise that running the tests is simpler than I thought * Update install instructions for developers * Update INSTALL to be consistent * Correct some failing tests * Update tests.rst (SciTools#3082) Added one line on how to test. This is mentioned in the Iris Check List in pulls.rst list, but would be good to have it here as well (useful for new contributors like me). * Add link to developer guide. (SciTools#3093) * Combine identical test routines for pcolor + pcolormesh. * Test 'contiguity_tolerance' use in iris.plot.pcolor and pcolormesh. * Unpin dask * Handle printing bounds of long time interval coords (SciTools#3140) * Handle printing time bounds * Whatsnew, whitespace fix for test result * iris.util.reverse on cubes (SciTools#3155) * make cube reversing official * review: test conventions, etc. * review: enable coord specification * add whatsnew * review: AssertRaises --> AssertRaisesRegexp * cube error handling * Improved Aux factory error handling: better message, delivered earlier (SciTools#3182) * Tests for expected behaviour * Raise error if aux factory relies on a coordinate external to cube * Include coordinate name in error message * Don't check whether dependencies that are "None" are in the coords. * Review changes: check cube name in error and variable rename * Truncated variable for flake8 * Correct cube collapse code * Provide info on pp-field indices in the file for structured um loads. (SciTools#2977) * Field locations for structured um loads. * Remove unused import. * Fix test for new implementation. * Added whatsnew. * Review changes. * simplify dev guide * Integration test for iris.plot.contour with 2d coords. (SciTools#3189) * Integration test for iris.plot.contour with 2d coords. * Review changes. * pinned numpy to not 1.15.2 in core requirements * Public contiguity checking (SciTools#3144) :champagne: :1st_place_medal: for sticking with it @corinnebosley * Populate whatsnew 2.2 (SciTools#3192) * started adding entries to whatsnew * more updates to whatsnew * first complete draft of 2.2 whatsnew * some small corrections * made corrections suggested in review * Update version str * Update CF standard name table to v59 * Fix docstring re structured text warnings (SciTools#3073) (SciTools#3197) * updated version string to 2.3.0dev0 * add contact options to README removed gitter from contact list * Add stash mapping changes to v2.1.0 changelog replaced list with link of stash mappings * expanded name of dev group * added an n for better flow * Fix broken link to dask docs (SciTools#3204)
This PR addresses the issue of the
imagehash
package releasing version4.0
, which addresses an error in the creation of hashes, see https://github.com/JohannesBuchner/imagehash#changelogidiff
imagerepo.json
andtest-iris-imagehash
repo.travis-ci
fortest-iris-imagehash
repoCloses #2973, #2974