Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure tests use ncgen/ncdump from the Python env; rationalise uses of ncgen. #4794

Merged
merged 7 commits into from
Jun 20, 2022
Merged

Ensure tests use ncgen/ncdump from the Python env; rationalise uses of ncgen. #4794

merged 7 commits into from
Jun 20, 2022

Conversation

pp-mo
Copy link
Member

@pp-mo pp-mo commented Jun 13, 2022 •
edited
Loading

This seemed desirable because I found that certain "assertCDL"s were working fine from the command-line but failing when run in PyCharm. The problems occurred regardless of whether running via unittest or PyTest.
It was using a really old fallback version of ncdump with slightly different flag behaviour, and output in some cases.
N.B. no problems ever occurred in CI -- this is purely a developer convenience problem !

This is because spawned processes don't inherit the path extensions you get from a 'conda activate'.
It quite probably applies to Eclipse too, since that also controls PATH/PYTHONPATH in non-obvious ways.

@pp-mo pp-mo requested a review from trexfeathers June 13, 2022 12:26
trexfeathers
trexfeathers requested changes Jun 13, 2022
View reviewed changes
Copy link
Contributor

@trexfeathers trexfeathers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pp-mo!

Any idea what to make of the test failures? They seem pretty opaque.

=================================== FAILURES ===================================
_______________________ lib/iris/tests/test_pickling.py ________________________
[gw6] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw6' crashed while running 'lib/iris/tests/test_pickling.py::TestPickle::test_cubelist_pickle'
_________________________ lib/iris/tests/test_plot.py __________________________
[gw8] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw8' crashed while running 'lib/iris/tests/test_plot.py::TestPcolormeshNoBounds::test_zx'
__________ lib/iris/tests/integration/analysis/test_area_weighted.py ___________
[gw1] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw1' crashed while running 'lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_real_save'
__________ lib/iris/tests/integration/analysis/test_area_weighted.py ___________
[gw0] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw0' crashed while running 'lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_lazy'
__________________ lib/iris/tests/unit/plot/test_contourf.py ___________________
[gw12] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw12' crashed while running 'lib/iris/tests/unit/plot/test_contourf.py::TestStringCoordPlot::test_xaxis_labels_with_axes'
__________ lib/iris/tests/integration/analysis/test_area_weighted.py ___________
[gw3] linux -- Python 3.8.13 /tmp/cirrus-ci-build/.nox/tests/bin/python
worker 'gw3' crashed while running 'lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_real_start'

lib/iris/tests/stock/netcdf.py Outdated Show resolved Hide resolved
lib/iris/tests/unit/fileformats/netcdf/test_load_cubes.py Outdated Show resolved Hide resolved
lib/iris/tests/integration/experimental/test_ugrid_save.py Outdated Show resolved Hide resolved
lib/iris/tests/integration/experimental/test_ugrid_save.py Outdated Show resolved Hide resolved
@rcomer
Copy link
Member

rcomer commented Jun 13, 2022

Any idea what to make of the test failures?

You’ve probably already noticed, but the crashing worker failures appear in several recent PRs.

@pp-mo
Copy link
Member Author

pp-mo commented Jun 13, 2022

Any idea what to make of the test failures?

You’ve probably already noticed, but the crashing worker failures appear in several recent PRs.

Well I hadn't, I'd so far only seen my own.
But from re-spinning, it is always the same tests that are failing.
In the other latest ones, I think we have "most" of the same ones failing ...

failure in #4791 in #4792 in #4794
lib/iris/tests/test_pickling.py::TestPickle::test_cubelist_pickle X X X
lib/iris/tests/test_plot.py::TestPcolormeshNoBounds::test_zx - - X
lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_lazy X X X
lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_real_save X X X
lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_real_start X X X
lib/iris/tests/integration/fast_load/test_fast_load.py::TestBasic__Iris::test_cross_file_concatenate X - -
lib/iris/tests/unit/analysis/maths/test_subtract.py::TestBroadcasting::test_collapse_zeroth_dim - X -
lib/iris/tests/unit/analysis/stats/test_pearsonr.py::Test::test_broadcast_cubes_weighted - X -
lib/iris/tests/unit/plot/test_contourf.py::TestStringCoordPlot::test_xaxis_labels_with_axes - - X

@pp-mo
Copy link
Member Author

pp-mo commented Jun 13, 2022

from re-spinning, it is always the same tests that are failing.
In the other latest ones, I think we have "most" of the same ones failing ...

Though from the messages, it appears to be like a timeout or resource failure ?
( but I haven't checked ALL the message yet )

The PyTest output is much harder to deal with than the older form >sigh<

@pp-mo pp-mo mentioned this pull request Jun 13, 2022
Merged
@pp-mo
Copy link
Member Author

pp-mo commented Jun 13, 2022 •
edited
Loading

I grabbed an older code version I previously used to debug a different problem.
The tests on that used to pass, but now have similar problems, when re-spun :
https://cirrus-ci.com/task/5930410508025856

So, it isn't to do with package versions or lockfiles, since that is still using older ones.
I suspect a resourcing issue with Cirrus or our account.

@lbdreyer also noted that where we have the usual orange block with the message "Not enough compute credits to prioritize tasks", in that block we are now also getting the ominous message "Container errored with 'OOMKilled'".
Sounds significant -- and I missed it up to now because of the spurious message you always get.

@rcomer
Copy link
Member

rcomer commented Jun 13, 2022

So, it isn't to do with package versions or lockfiles, since that is still using older ones

Could something have changed in the linux environment? I’m thinking of #3993, where we started getting failures because glibc changed.

@pp-mo pp-mo mentioned this pull request Jun 14, 2022
Closed
@pp-mo pp-mo marked this pull request as draft June 14, 2022 16:50
@pp-mo
Copy link
Member Author

pp-mo commented Jun 14, 2022

NOTE: made draft, as I want to include changes paralleling #4800, which is not yet merged, to get tests passing.
So this will now need a rebase before we're done.

@trexfeathers
Copy link
Contributor

So this will now need a rebase before we're done.

Now's the time 👍

@pp-mo
Copy link
Member Author

pp-mo commented Jun 17, 2022

So this will now need a rebase before we're done.

Now's the time +1

For starters, I've removed the last commit, which contained the cirrus hack.

It no longer conflicts
We'll just see how that goes ...

pp-mo
pp-mo commented Jun 17, 2022
View reviewed changes
lib/iris/tests/stock/netcdf.py Outdated Show resolved Hide resolved
@pp-mo
Copy link
Member Author

pp-mo commented Jun 17, 2022 •
edited
Loading

@trexfeathers OK we have problems with the CI that I don't understand.
Can you advise ? [[ EDITED: sorry, I think we know what is happening ]]

We think that #4809 may fix ?

@pp-mo
Copy link
Member Author

pp-mo commented Jun 17, 2022

We now have "both kinds of test" here !
I expect a rebase is desirable, but right now pausing for #4809
(still in Draft)

@pp-mo
Copy link
Member Author

pp-mo commented Jun 17, 2022

Rebased.
Nearly there I hope

@pp-mo pp-mo marked this pull request as ready for review June 17, 2022 11:48
@pp-mo
Copy link
Member Author

pp-mo commented Jun 17, 2022

Good to go, I think.
Over to you @trexfeathers

@trexfeathers
Copy link
Contributor

Looking good @pp-mo. This deserves a What's New entry

@pp-mo
Copy link
Member Author

pp-mo commented Jun 20, 2022

hi @trexfeathers
Good to go now on this ? 🙏

@trexfeathers
Copy link
Contributor

hi @trexfeathers Good to go now on this ? 🙏

#4794 (comment)

@trexfeathers trexfeathers merged commit af07045 into SciTools:main Jun 20, 2022
@pp-mo
Copy link
Member Author

pp-mo commented Jun 20, 2022

Many thanks @trexfeathers !

@pp-mo pp-mo mentioned this pull request Jun 20, 2022
Merged
@pp-mo pp-mo deleted the fix_test_spawns branch June 20, 2022 17:56
lbdreyer added a commit to lbdreyer/iris that referenced this pull request Sep 14, 2022
@lbdreyer @bjlittle
@github-actions @web-flow @trexfeathers @dependabot @rcomer @pre-commit-ci @pp-mo @wjbenfold @dennissergeev @tkknight @schlunma @stephenworsley @scitools-ci @krikru @ESadek-MO
Qa issue (#9)
edd6c33 
* 3.2 version and whats new (#4559)

* Reset whats new (#4563)

* reset whats new

* Specify minor in whats new

* update trove classifiers (#4564)

* Update version to 3.3.dev0 (#4565)

* Updated environment lockfiles (#4567)

Co-authored-by: Lockfile bot <[email protected]>

* New tool-agnostic ASV environment management (#4571)

* New tool-agnostic ASV env management.

* Benchmarking only build the latest Python.

* Increase benchmark accuracy by increasing rounds.

* Fix ASV rounds mistake.

* ASV clearer use of _interpolate_commands.

* adopt dependabot GHA (#4568)

* Bump peter-evans/create-pull-request from 3.8.2 to 3.12.1 (#4577)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3.8.2 to 3.12.1.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/052fc72b4198ba9fbc81b818c6e1859f747d49a8...f22a7da129c901513876a2380e2dae9f8e145330)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/stale from 4.0.0 to 4.1.0 (#4575)

Bumps [actions/stale](https://github.com/actions/stale) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/v4.0.0...v4.1.0)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/script from 4 to 5.1.0 (#4576)

* gha: lockfiles labels and auto-pr details (#4578)

* Fix refresh lockfile worrkflow pull request title (#4579)

* New tool-agnostic ASV environment management (#4571)

* New tool-agnostic ASV env management.

* Benchmarking only build the latest Python.

* Increase benchmark accuracy by increasing rounds.

* Fix ASV rounds mistake.

* ASV clearer use of _interpolate_commands.

* Fix load_http bug, extend testing, and note to docs (#4580)

* Fix opendap bug, add docs and extra testing

* Add whats new entry

* Update docs/src/whatsnew/3.2.rst

Co-authored-by: Bill Little <[email protected]>

* Add warning box

Co-authored-by: Bill Little <[email protected]>

* Loading Benchmarks (#4477)

* Synthetic FF PP NetCDF and loading benchmarks.

* fix test (#4585)

* [pre-commit.ci] pre-commit autoupdate (#4587)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/psf/black: 21.12b0 → 22.1.0](https://github.com/psf/black/compare/21.12b0...22.1.0)
- [github.com/asottile/blacken-docs: v1.12.0 → v1.12.1](https://github.com/asottile/blacken-docs/compare/v1.12.0...v1.12.1)
- [github.com/aio-libs/sort-all: v1.1.0 → v1.2.0](https://github.com/aio-libs/sort-all/compare/v1.1.0...v1.2.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Finalise whatsnew and version string update (#4588)

* docs linkcheck skip (#4590)

* Add missing commit to v3.2.x and update version number (#4593)

* fix trove classifier (#4324)

* update version to 3.2. post release

Co-authored-by: Bill Little <[email protected]>

* Bump actions/script from 5.1.0 to 6 (#4586)

Bumps [actions/script](https://github.com/actions/script) from 5.1.0 to 6.
- [Release notes](https://github.com/actions/script/releases)
- [Commits](https://github.com/actions/script/compare/v5.1.0...v6)

---
updated-dependencies:
- dependency-name: actions/script
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Yaml fixes + clarifications. (#4594)

* Yaml fixes + clarifications.

* Update .github/workflows/stale.yml

Co-authored-by: Ruth Comer <[email protected]>

Co-authored-by: Bill Little <[email protected]>
Co-authored-by: Ruth Comer <[email protected]>

* Overnight benchmarks (#4583)

* Utility class in netcdf loader should not be public. (#4592)

* Utility class in netcdf loader should not be public.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename container for better clarity.

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Stop using nc_time_axis.CalendarDateTime (#4584)

* stop using CalendarDateTime

* update dependencies

* stronger tests for _fixup_dates

* add whatsnew

* Update 3.2.rst whats new with section for bug fixes (#4604)

* Bump peter-evans/create-pull-request from 3.12.1 to 3.13.0 (#4607)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3.12.1 to 3.13.0.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/f22a7da129c901513876a2380e2dae9f8e145330...89265e8d24a5dea438a2577fdc409a11e9f855ca)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Support false-easting and false-northing when loading Mercator-projected data (#4524)

* Add extra pieces to get false easting and northing

* tests

* Add extra test file ref and cml

* lib/iris/tests/results/netcdf/netcdf_merc_false.cml

* Update cml for new test data

* Bump test data version for cirrus

* What's new

* Bump peter-evans/create-pull-request from 3.13.0 to 3.14.0 (#4608)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3.13.0 to 3.14.0.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/89265e8d24a5dea438a2577fdc409a11e9f855ca...18f7dc018cc2cd597073088f7c7591b9d1c02672)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert plotting-vs-y (#4601)

* revert plotting-vs-y

* whatsnew

* Bump actions/stale from 4.1.0 to 5 (#4612)

Bumps [actions/stale](https://github.com/actions/stale) from 4.1.0 to 5.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/v4.1.0...v5)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/checkout from 2 to 3 (#4611)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Use crs.globe in _crs_distance_differentials() (#4605)

* Use crs.globe in _crs_distance_differentials(), add a relevant test, update whatsnew

* add a ref to the issue and PR

* add dennis (#4614)

* Final offline benchmark migration (#4562)

* Overnight benchmarks remove ambiguity between file and commit names. (#4620)

* purge deploy key (#4615)

* Remove no_clobber task from Refresh lockfiles Action (#4618)

* Remove no_clobber task from Refresh lockfiles Action

* review actions - add extra infor to pr message

* Scalar Scatter Plot (#4616)

* add failing test

* pass test

* initial review actions

* test 2nd arg scalar

* whatsnew

* Updated environment lockfiles (#4624)

Co-authored-by: Lockfile bot <[email protected]>

* Overnight benchmarks - find a valid issue assignee (#4627)

* Overnight benchmarks find a valid issue assignee.

* Overnight benchmarks safer check for a valid issue assignee.

* Votable Issues (#4617)

* Create test2.yml

* gha test

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* Update refresh-votable-issues.yml

* initial

* tidy

* updated whatsnew

* removed defunct ext links

* Update docs/src/votable_issues.rst

Co-authored-by: Ruth Comer <[email protected]>

* added suggestion to subscribe to voted issues.

* Renamed votable to voted.  Layout tweaks to the voted issues page.

Co-authored-by: Ruth Comer <[email protected]>

* Avoid dimensionality mismatch between data array and CF variables. (#4610)

* Sperf & Cperf Benchmarks (#4621)

* Update version to 3.2.1 and fix whatsnew (#4635)

* Improve Small Benchmark Accuracy (#4636)

* Remove another memory benchmark that's smaller than the noise level.

* Fixed import benchmarks.

* Updated GHA frequency and table name for Voted Issues (#4641)

* updated gha frequency and table name

* added new ignore for linkcheck

* use conda-lock -k explicit (#4644)

* Bump actions/cache from 2 to 3 (#4650)

Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump peter-evans/create-pull-request from 3.14.0 to 4 (#4652)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3.14.0 to 4.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/18f7dc018cc2cd597073088f7c7591b9d1c02672...d6d5519d05f5814158ef015b8448f2f74648c421)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix aggregated_by mdtol/masked constant problem (#4246)

* add failing test

* pass test

* whatsnew

* remove redundant copy call

* Make iris.tests.stock.simple_1d respect bounds arg (#4658)

* Make iris.tests.stock.simple_1d respect bounds arg

* What's new

* What's new

* Improved check for valid user in overnight BM run. (#4659)

* Preserve position when replacing Axes with GeoAxes (#4273)

* preserve position

* add test

* comment to clarify test

* whatsnew

* use original position

* Lazy Percentile Aggregator (#3901)

* implement lazy percentile

* fix map_complete_blocks tests

* fix legacy masked type no mask tests

* add test for map_complete_blocks on dask array

* handle axis expressed as sequence

* fix docstring

* add lazy_aggregate method and mdtol handling

* reshape for multi-dim collapse

* mdtol handling for added trailing dimension

* update aggregator instance docstring

* simplify array reshape

* factor out array reshaping as decorator

* reduce repetition in tests

* PERCENTILE multi-axis tests

* add tests for array wrangling

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* py3.7

* test_PERCENTILE.py tweaks

* cut down post_process tests

* clarify wrapper tests

* whatsnew

* revert py3.7 change

* Update lib/iris/analysis/__init__.py

Co-authored-by: Will Benfold <[email protected]>

* review: remove docstring kwarg; define _real_percentile

* review: blank line for note

* review: remove scalar_percent check

* review: test__axis_to_single_trailing.py

* review: simplify structure

* review: _percentile docstring args

* review: remove axis description from _percentile docstring

* review: don't use rollaxis

* review: generalize LazyMixin to also work for non-lazy.  Rename

* review: move Test_lazy_aggregate tests into MaskedCalcMixin

* review: generalise MaskedCalcMixin

* review: remove stray as_lazy_data calls

* review: use mixins for Test_aggregate

* review: move multi axis tests to their own mixin

* review: use MultiAxisMixin in both lazy classes

* review: add tests for fast real aggregation

* docstrings

* review: test clarity tweaks

* review: test alphap, betap passed to scipy

* review: use assertEqual

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Will Benfold <[email protected]>

* Fix nearest_neighbour_index for edge case where requested point is float and bounds are int (#4245)

* add failing test

* pass test

* add whatsnew

* Include note to strip out Artifactory (#4666)

* Include note to strip out Artifactory

* What's new

* update black (#4668)

* Fixed cube arithmetic for cubes with meshes. (#4651)

* Use separate cml files for tests/unit/analysis/maths/test_slice multiple passes.

* Roughly functional, but lots of tests to fix.

* Fix

* Fix mockist test for new-style Resolve.  Mesh support not yet tested.

* Fix _create_prepared_item when points/bounds passed in.

* --no-edit

* Simplify prepared-item usage.

* Added tests

* Added whatsnew.

* Reference Iris issues for outstanding problems.

* Small simplifications.

* Updated statement on cube arithmetic with meshes.

* Fix use of 'MathsAddOperationMixin'; add coord-mismatch tests on meshcubes.

* Review changes.

* Hack test which was failing.

* Updated environment lockfiles (#4655)

Co-authored-by: Lockfile bot <[email protected]>

* 'Deep' benchmarks to catch scaling with field count (#4654)

* 'Deep' benchmarks to catch scaling with field count

* Better sized benchmarks

* What's new

* Update benchmarks/benchmarks/load/__init__.py

Co-authored-by: Martin Yeo <[email protected]>

Co-authored-by: Martin Yeo <[email protected]>

* Automatic reversal of DimCoord bounds (#4466)

* first pass at ensuring contiguous bounds

* reduce scope and fix old tests

* bounds setter tests

* test contiguity preserved

* check bounds within reverse tests

* add comment; merge test

* simpler approach

* revert now redundant changes

* update cml

* cell equality to ignore bound order

* update cml

* move merge test to integration

* review: simple actions

* review: split up cube reverse tests

* review: check bounds in all cube reverse tests

* whatsnew

* review: reduce test repetition

* tweak whatsnew

* blank line

* Updated environment lockfiles (#4677)

Co-authored-by: Lockfile bot <[email protected]>

* Updated environment lockfiles (#4678)

Co-authored-by: Lockfile bot <[email protected]>

* Reorder Constraints sections of user guide, add time bounds example (#4656)

* move constraint detail to subsetting cube, add example of constraining bound

* Add whats new

* Bump peter-evans/create-pull-request from 4.0.0 to 4.0.1 (#4675)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/d6d5519d05f5814158ef015b8448f2f74648c421...f1a7646cead32c950d90344a4fb5d4e926972a8f)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* tweak whatsnew (#4682)

* Implement Pydata theme (#4661)

* new theme baseline

* update

* added discussion directive and whatsnew

* fixed bullet point indentation

* updated NOX_CACHE_BUILD to force update the nox cache

* added pr ref

* Update lock-file generation in response to conda-lock change

* Make lockfile shareable and bump nox cache

* MAin branch updates to latest.rst (from dev.rst)

* change references of dev.rst to latest.rst

* corrected bullet points indentation to render correctly.

* updated

* update whatsnew

* Review action and fixed typo for link

* update lock

* updated from main

Co-authored-by: Will Benfold <[email protected]>
Co-authored-by: Bill Little <[email protected]>

* Update percentile tests (#4684)

* update percentile tests

* review action

* Updated environment lockfiles (#4685)

Co-authored-by: Lockfile bot <[email protected]>

* Bump actions/download-artifact from 2 to 3 (#4686)

Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/upload-artifact from 2 to 3 (#4687)

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump peter-evans/create-pull-request from 4.0.1 to 4.0.2 (#4688)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4.0.1 to 4.0.2.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/f1a7646cead32c950d90344a4fb5d4e926972a8f...bd72e1b7922d417764d27d30768117ad7da78a0e)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add dynamic matrix to refresh lockfile job (#4573)

* Add dynamic matrix to refresh lockfile job

* review actions

* added link check and ensured https use (#4694)

* [pre-commit.ci] pre-commit autoupdate (#4691)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.1.0 → v4.2.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.1.0...v4.2.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Comment rendering (#4689)

* used numpy docstrings to fix docs rendering

* comment rejig

* apply numpydoc strings

* corrected notes

* updated docstring guidance.

* added explicit mention of numpydoc strings

* added whatsnew

* added whatsnew PR ref

* Update lib/iris/fileformats/nimrod_load_rules.py

Co-authored-by: Will Benfold <[email protected]>

* Update docs/src/whatsnew/latest.rst

Co-authored-by: Will Benfold <[email protected]>

* review actions.

Co-authored-by: Will Benfold <[email protected]>

* Updated environment lockfiles (#4696)

Co-authored-by: Lockfile bot <[email protected]>

* Cache regular points generation (#4698)

* Cache regular points generation

* Move func to util

* Just apply lru_cache to the function once

* What's new

* Git ignore the benchmarks data folder (#4699)

* Git ignore the benchmarks data folder

* Update path and description

* Catch `AttributeError` from removing `CFReader._dataset` when it doesn't exist (#4646)

* Implement full Mercator parameter support (#4609)

* Changes to coord_systems

* Finish feature

* Define both standard_parallel and scale_factor... but leave one None

* Initial test changes

* Test updates and additions

* New expected test outputs

* What's new and test data version bump

* Review fix

* Edit the right file

* Updated environment lockfiles (#4706)

Co-authored-by: Lockfile bot <[email protected]>

* Add evertrol to whatsnew of #4646 given #4705 (#4709)

* Add `iris.analysis.MAX_RUN` aggregator (#4676)

* Failing tests

* Start with the tests

* Draft implementation

* Test results

* Account for masked data and multiple axes

* Testing, bug fixes

* Reduce duplication

* Update .cml and fix test failures

* Don't use map_complete_blocks

* Use dask versions of functions

* Test and fix masked data handling

* Use da.from_array instead of as_lazy_data

* Test chunking, fix dask.array import

* Restore _lazy_data.py as we're not using those changes

* credit Will's epic review (#4711)

* add bullet (#4715)

* Weighted `aggregated_by` (#4589)

* First suggestion for weighted aggregated_by

* Added lazy weighted aggregation

* Added first tests for weighted aggregated_by

* Added further tests and allowed returned kwarg

* Added unit tests for weighted aggregated_by

* Moved _get_aggregator_fn to iris.analyis

* Reused 'returned' keyword

* Added test for lazy output if 'returned' is used

* Removed checksum=False from tests

* Expanded test for lazy aggregation since PERCENTILE is lazy now

* Added schlunma to common links

* Addaux speedup (#4718)

* Attempt faster add-aux-fact

* Added whatsnew draft.

* Put PR ref into whatsnew.

* Review changes.

* Remove time_return benchmarks. (#4717)

* Credit reviewer in what's new contributions. (#4705)

* Credit reviewer in what's new contributions.

* Correct example link in what's new docs.

Co-authored-by: Will Benfold <[email protected]>

Co-authored-by: Will Benfold <[email protected]>

* Change Cell equality to equate nans (fixes #4681) (#4701)

* nans in cells satisfy equality

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address review comments

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Netcdf saving: workaround known netcdf bug. (#4407)

* Netcdf saving: workaround known netcdf bug.

* Fix netcdf dim renaming for anonymous dims.

* Fixes to test-mesh generation to preserve save-load roundtrips.

* Added tests for all the different places where code was fixed.

* Tidy some formatting and comments.

* Fix rebase error?

* Review changes.

* Updated environment lockfiles (#4724)

Co-authored-by: Lockfile bot <[email protected]>

* MAX_RUN into docs (#4726)

* Datum support in Iris (#4704)

* Support in helpers

* coord_systems changes

* More additions

* Revert some FUTURE guards and condense datum application

* Fixing tests and improving

* More additions

* Tests pass

* transform_points test

* Review fixes

* In datums, unknown means None

* Fix test

* Suppress FutureWarning in the tests that were failing due to it

* Review fixes

* What's new

* Extra test result

* Bump peter-evans/create-pull-request from 4.0.2 to 4.0.3 (#4733)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4.0.2 to 4.0.3.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/bd72e1b7922d417764d27d30768117ad7da78a0e...f094b77505fb89581e68a1163fbd2fffece39da1)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updated environment lockfiles (#4732)

Co-authored-by: Lockfile bot <[email protected]>

* Replace nose with pytest (#4734)

* Replace nose with pytest

* Reinstate magic processor number

* add psutil, fstrings and rename TestMixin

* Caching during load (#4716)

* Caching

* Move _optimum_chunksize docstring

* Should be copying coords because they're mutable

* Handle failure of datetime comparisons gracefully

* Review fixes and more careful checks on mutable objects

* coord_system doesn't have 'copy' but __repr__ covers the whole thing (I think)

* Review fixes

* Revert coord_system changes

* Still need to handle tuple issue

* What's new

* Fix prompted by test failure

* Limit cache size

* Work around comparable datetime issue

* More sensible climatological time points (#4723)

* Initial moves to introduce result_coord_func

* climatological keyword

* Climatology argument to aggregated_by

* Fix the tests

* Review changes from RC

* Test fixes

* Further test fixes

* Move test code back out after rebase

* Different fix for self.coord_dims failure

* Add ifmain

* Revert inadvertent doctest changes

* Review improvements

* Fix doctest black broke

* Stop new tests being executable, which makes the test runner miss them

* Docstrings to comments so they don't obscure test names in results

* Apply suggestions from code review

Co-authored-by: Ruth Comer <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Review fixes

* What's new

Co-authored-by: Ruth Comer <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove Mule monkey patch. (#4738)

* climatology doc tweak (#4740)

* Mesh docs - include examples of converting different mesh formats. (#4739)

* Mesh docs - include examples of converting different mesh formats.

* SMC mesh example fix gotcha.

* Add sample images to mesh conversion examples.

* Mesh conversion examples - fix bullet points.

* Mesh conversion examples - print meshes.

* Whats New entry.

* Mesh conversion examples - more detailed comments.

* Gallery: update NEMO example (#4741)

* update nemo example

* clearer comment

* fixed numbered list entry (#4743)

* Plotting circular data extends args as well as the data (#4649)

* Extend any kwarg array

* Add test

* Add comments and update imagerepo.json

* Try using the value as an array, in case it's been given as a list etc.

* What's new

* Datum performance improvements (#4731)

* Set stacklevel to get more useful FutureWarning

* Remove spare printing

* Only generate crs and globe on demand, handle attr setting

* Updates docstrings to make it clearer what's going on

* Additional tests (and fixes found from testing)

* Warn that inverse_flattening setting doesn't help

* Updated environment lockfiles (#4742)

Co-authored-by: Lockfile bot <[email protected]>

* Extra datum translation (#4753)

* Updated environment lockfiles (#4760)

Co-authored-by: Lockfile bot <[email protected]>

* Permit fast percentile with mdtol=0 (#4755)

* Permit fast percentile with mdtol=0

* What's new

* Check arrays like they're masked, and make the expected arrays masked

* Bigger arrays to avoid np quirk, bug fix, check masking in 2d_multi

* Suppress numpy masked percentile warning

* Use a larger test array that numpy handles how it handles most of them (the worse case)

* Broadcast the mask in the non-lazy case with added dim

* Bug fix on ndim test

* Review fixes

* Credit @rcomer on PR #4755 review (#4761)

* CPerf and SPerf Benchmark Fixes (#4758)

* CSPerf fixes.

* Correct argument hashing in file names.

* Warn if BENCHMARK_DATA is not set.

* SPerf combine_regions better file handling.

* SPerf don't store FileMixin files long term - too much space.

* More realistic SPerf file sizes, allow larger-than-memory files.

* Add SPerf comment about total disk space.

* Corrected CSPerf directories printout.

* Remove namespace conflict in SPerf combine_regions.

* don't use np.float or np.int (#4763)

* Test warnings (#4764)

* import from collections.abc

* deprecated csr module

* unittest deprecations

* random_integers -> randint

* collections again

* inspect deprecation

* reduce redunancy

* non test class

* Increase speed of iris.analysis.trajectory.interpolate with linear interpolation (#4366)

* Interpolate all points at once rather than sequentially

* This splits the internals of cube.interpolate in an effort not to make so much superfluous data, but is slow

* Generate the extra values then discard them, but with minimal looping

* Updated docstrings to clarify behaviour

* Benchmarks for trajectory (linear and nearest neighbour)

* Update error thrown for aux coord over interpolated and uninterpolated dimensions, and add unit tests

* Update cml

* What's new has moved

* What's new fix for rebase

* Prevent truncation of cell method descriptions with nested brackets (#4436)

* Add failing test for brackets within comments

* Fix test

* Separate private function now splits the cell method before it's parsed

* Add additional failing test for multiple axis methods

* Handle multiple axis cell methods

* Update docstring to reflect eventual choices

* Add warnings to function, and test them

* Significantly simpler warning checks

* Developer documentation on available testing conveniences (#4600)

* Developer documentation on available testing conveniences

* what's new

* Update to be easily usable without code links working

* Rebase

* Cubelist contain only cubes -- resurrected (#4767)

* ensure cubelists only contain cubes

* address __iadd__ and __setitem__

* __setitem__ tests

* test for setting more than 1 item

* Fix __setitem__ and Py2 tweaks

* implement __setslice__ for Python2.7

* change exceptions to warnings

* stickler

* duck type check; move helpers outside class

* blank lines

* proposed: revert warnings to exceptions

* remove stray extra 'test_fail'

* pass sequences through __init__; _assert_is_cube

* Replace CubeList init with new. Cube testing duck-type.

* Review change.

* Provide new-style whatsnew.

* Fix whatsnew

Co-authored-by: Ruth Comer <[email protected]>

* Updated environment lockfiles (#4775)

Co-authored-by: Lockfile bot <[email protected]>

* Redirect whatsnew to latest (#4727)

* Redirect whatsnew

* Get the top bar working too

* Clean build with right links

* Update release docs

* Review fixes

* update section name for clarity

* Bump peter-evans/create-pull-request from 4.0.3 to 4.0.4 (#4783)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4.0.3 to 4.0.4.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/f094b77505fb89581e68a1163fbd2fffece39da1...923ad837f191474af6b1721408744feb989a4c27)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updated environment lockfiles (#4782)

Co-authored-by: Lockfile bot <[email protected]>

* Fix whatsnew (#4784)

* Fix whatsnew

* Instead of changing 3.0.rst, rename 'whatsnew.rst' to 'index.rst'

* Missed one

* `PolarStereographic` implementation (#4773)

* PolarStereographic implementation

* Test data version

* What's new

* Accept suggestion to lib/iris/coord_systems.py

Co-authored-by: Patrick Peglar <[email protected]>

* Permit Stereographic saving

* Update what's new

* Review fixes

Co-authored-by: Patrick Peglar <[email protected]>

* Add mesh details to cube printout. (#4778)

* Slightly refactor repr-html code to rationalise section handling.

* Test repr-html of string attributes.

* Add mesh details to cube printout.

* Update meshcube printouts in docs.

* Add Mesh section to repr-html.

* Review change : make test clearer.

* Review change : fix string truncation/quoting.

* Retire use of the New labels. (#4801)

* pinned pydata-sphinx-theme until the docs are dark mode friendly. (#4796)

* adopt ci gha (#4503)

* adopt ci gha

* bump iris-test-data to latest version

* ci noxify

* remove dummy cache key name

* remove precommit nox task

* added whatsnew entry

* remove mpl cfg for linkcheck

* trial composite nox-cache

* composite iris-data-cache action

* composite conda-env-cache action

* composite conda-pkg-cache action

* composite cartopy-cache action

* required lock-file composite action input

* review action

* review action

* nox inputs.lock_file

* env_name composite action input

* replace HOME with tilde usage

* cancel in-progress workflow jobs

* refresh-lockfiles gha note

* Scripted Logo Generation, take two (#3935)

* Added logo generation Python script.

* Replace lambda scale_func.n

* Matched current logo appearance.

* Simplified gradient code.

* Renamed land_mask to land_clip.

* Make new svg instead of faffing with reformatting existing.

* Refactor out 'style' in attributes.

* File header.

* Integrated logo generation script into Sphinx docs generation.

* Environment variable logo names.

* Removed banner width calculation to avoid font dependency.

* Refactored to also generate rotating logos.

* Replace plt.tightlayout with plt.subplots_adjust.

* Replaced OrderedDicts with dicts (not needed >=Py3.6).

* Numpy standard import form.

* Replaced animated SVG with precusor files for animated GIF.

* Corrected logo URL to source from ReadTheDocs site.

* Changes to better match 'hand-created' logo.

* Fixed logo rotate zip title.

* Relocated logo generation script to SciTools/marketing.

* update links (#3942)

* update links

* added s to http

* Added logo generation Python script.

* Replace lambda scale_func.n

* Matched current logo appearance.

* Simplified gradient code.

* Renamed land_mask to land_clip.

* Make new svg instead of faffing with reformatting existing.

* Refactor out 'style' in attributes.

* File header.

* Integrated logo generation script into Sphinx docs generation.

* Environment variable logo names.

* Removed banner width calculation to avoid font dependency.

* Refactored to also generate rotating logos.

* Replace plt.tightlayout with plt.subplots_adjust.

* Replaced OrderedDicts with dicts (not needed >=Py3.6).

* Numpy standard import form.

* Replaced animated SVG with precusor files for animated GIF.

* Corrected logo URL to source from ReadTheDocs site.

* Changes to better match 'hand-created' logo.

* Fixed logo rotate zip title.

* Relocated logo generation script to SciTools/marketing.

* Include version string in logo generation.

* Reference official logo generation code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix README docs reference.

* Include the logos, don't generate on the fly.

Co-authored-by: Ruth Comer <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix composite gha (#4809)

* fix composite gha

* bump gha cache_build to force purge

* Whats new entry for SVG logo. (#4810)

* Updated environment lockfiles (#4815)

Co-authored-by: Lockfile bot <[email protected]>

* add gha workflow badges (#4813)

* address npy disable cpu features warning (#4811)

* Ensure tests use ncgen/ncdump from the Python env; rationalise uses of ncgen. (#4794)

* Ensure tests use ncgen/ncdump from the Python env; rationalise uses of ncgen a bit.

* Tidy some imports.

* Rationalise use of ncgen and avoid subprocess 'shell=True'.

* Review changes: revert to importing iris.tests first.

* Tidy routines slightly.

* Fix subprocess call.

* Added whatsnew.

* Limit memory benchmark result to 5Mb minimum (#4818)

* docs: cirrus-ci to gha (#4821)

* docs: cirrus-ci to gha

* review actions

* Stereographic load with scale factor (#4785)

* Allow passing of method keyword to numpy.percentile (#4791)

* allow passing of method keyword to numpy.percentile

* review actions

* pass all kwargs

* fix doc kwargs

* Update image testing (#4759)

* Use images from test data

* Bug fixes and docs

* Include circular plot image

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Oops, missed it. There now

* Update test data version

* Update __all__

* Type hinting

* No underscore

* Join strings

* Other underscores

* Other test data version location

* Doc fixes

* Imagerepo refresh info

* Move the lock

* Fix refs

* Remove image index (as it's now in the test data index)

* Fix the json test

* Revert pillow pin removal to separate concerns

* Update imagerepo for test that passes with penultimate val

* Second imagerepo update

* Third set of imagerepo changes

* Fourth imagerepo changes (gallery test)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* whatnew for #4791 (#4823)

* bump ci to latest iris-test-data version (#4825)

* auto generate gha auth token (#4817)

* auto generate gha auth token

* add docs for github app config

* linkcheck ignore metarelate.net

* remove metarelate URL whatsnew reference

* Fix stock CDL headers for unlimited time dimension. (#4827)

* Fix CDL headers for tests.stock.netcdf.

* Whats new entry.

* Improved unlimited dimension check in stock NetCDF generation.

* Whatsnew minor fix (#4829)

* Update latest.rst

* PR link

* add issue link

* Updated environment lockfiles (#4831)

Co-authored-by: Lockfile bot <[email protected]>

* rationalise doctests gha (#4819)

* rationalise doctests gha

* update job name with session

* add nox session as composite input

* minor tweak

* review action - tidy nox composite inputs

* Remove pin, update hashes, improve script (#4826)

* Remove pin, update hashes, improve script

* What's new

* Lockfile and whatsnew

* Shareable urls

* Later lockfile

* Shareable...

* Check for consistency before fiddling

* Updates for new test data (idiff'ed)

* Keep lockstep between imagerepo and data

* idiff always runs after imagerepo.json has been updated anyway

* Gallery test updates (also idiff'd)

* Update test data version

* Implement `iris.plot.fill_between` (#4647)

* Remove pin, update hashes, improve script

* What's new

* Lockfile and whatsnew

* Shareable urls

* Later lockfile

* Shareable...

* Check for consistency before fiddling

* Updates for new test data (idiff'ed)

* Keep lockstep between imagerepo and data

* idiff always runs after imagerepo.json has been updated anyway

* Gallery test updates (also idiff'd)

* Implement fill_between and write tests

* Simplify error check

* Test images

* What's new

* Image test results

* Fix error message

* Pre-emptive test data version update

Co-authored-by: Bill Little <[email protected]>

* Python 3.9 CI & Combine CI workflows (#4832)

* combine ci workflows

* remove badge

* ci unpin python

* post #4759 review actions (#4833)

* pytest less verbose (#4837)

* Remove redundant testing assert methods (#4838)

* remove assertWarnsRegexp

* remove assertStringEqual

* post rebase tidyings

* whatsnew

* NetCDF Generation Adjustments (#4836)

* remove pillow pin from setup.cfg (#4839)

* Support python 3.10 (#4840)

* extend ci support for py310

* fix Cell.__hash__ py310 nan behaviour

* review actions

* update github app docs (#4844)

* Lockfile update and Gregorian to Standard calendar change (#4847)

* sed gregorian

* sed CALENDAR_GREGORIAN

* rename cml files

* revert whatsnew 2.1 and test_cf.py

* revert spelling_allow

* post rebase sed 'gregorian'

* Updated environment lockfiles

* cf-units minimum pin

* review actions

Co-authored-by: Lockfile bot <[email protected]>

* Updated environment lockfiles (#4850)

Co-authored-by: Lockfile bot <[email protected]>

* avoid setuptools deprecation (#4848)

* [pre-commit.ci] pre-commit autoupdate (#4851)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.2.0 → v4.3.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.2.0...v4.3.0)
- [github.com/psf/black: 22.3.0 → 22.6.0](https://github.com/psf/black/compare/22.3.0...22.6.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Alternative Cell.__hash__ implementation for NaN support (#4852)

* Alternative Cell.__hash__ implementation for NaN support

* add whatsnew entry

* whatsnew update (#4853)

* adopt setuptools-scm for auto versioning (#4841)

* adopt setuptools-scm for auto versioning

* update py310.yml requirements

* update locks

* simplify docs version

* update dev release docs

* update .gitignore for vscode

* update locks

* add exclude _version.py to MANIFEST.in

* review actions

* update lock files

* update pyproject.toml

* what a whatsnew entry

* fix rtd version (#4855)

* Updated environment lockfiles (#4858)

Co-authored-by: Lockfile bot <[email protected]>

* pytest verbosity (#4861)

* Updated environment lockfiles (#4867)

Co-authored-by: Lockfile bot <[email protected]>

* Recommend `pytest` in developer guide (#4866)

* recommend pytest

* pros and cons

* tweaks

* add missing skips (#4865)

* Updated environment lockfiles (#4873)

Co-authored-by: Lockfile bot <[email protected]>

* Replace testing helper function (#4878)

* replace unmasked_data_as_1d_array

* whatsnew

* Cube arithmetic array type handling (#3790)

* add new tests; fix existing tests

* force lhs lazy if rhs lazy

* black

* add in-place tests

* add handling for unmasked, masked real array combo

* add tests for false skeleton

* rearrange _math_op_common if-loops

* revert maths.py to upstream/master

* simpler solution

* reduce masking arithmetic tests

* expand and improve tests

* add whatsnew

* fix rebase snafu

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update following numpy#21977

* Revert "update following numpy#21977"

This reverts commit 7d62ad6b8779129b9f4c2f7d26f070f5402f95e9.

* update comment

* a before b

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* automate sdist and wheel build and publish (#4849)

* automate sdist and wheel build and publish

* update names

* tidy package manifest and discovery

* add note to dev docs

* update MANIFEST.in

* Update .github/workflows/ci-wheels.yml

Co-authored-by: Martin Yeo <[email protected]>

* add unshallow comment to rtd

* reinstate docs release

* add pypa build comment

* test wheel

* added whatnew + tweak nox wheel emsgs

* Update docs/src/developers_guide/release.rst

Co-authored-by: Martin Yeo <[email protected]>

* Warn users of increased `recombine_submeshes` memory demand. (#4881)

* Warn users of increased recombine_submeshes memory demand.

* Clearer what's new entry.

* Sphinx reset modules (#4885)

* fix sphinx-gallery

* change gallery index reference

* Updated environment lockfiles (#4884)

Co-authored-by: Lockfile bot <[email protected]>

* [pre-commit.ci] pre-commit autoupdate (#4886)

updates:
- [github.com/PyCQA/flake8: 4.0.1 → 5.0.2](https://github.com/PyCQA/flake8/compare/4.0.1...5.0.2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix percentile docstring (#4888)

* [pre-commit.ci] pre-commit autoupdate (#4895)

updates:
- [github.com/PyCQA/flake8: 5.0.2 → 5.0.4](https://github.com/PyCQA/flake8/compare/5.0.2...5.0.4)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Upgrade `iris.util.mask_cube` (#4889)

* rewrite mask_cube

* fix import problems

* simplify _unmask_mask and its tests

* parametrize not in place 1d _mask_array

* pytest in-place _mask_array tests

* pytest _mask_array broadcasting tests

* remove iris.tests from test__mask_array

* attempt to preserve metadata

* pull masked array assertions off test class

* pytestify test__unmask_mask.py

* remove _unmask_mask; tweak test

* numpydoc docstring

* add CML checks

* preserve cube.metadata; add final CML

* path handling

* get hold of _mask_array result

Co-authored-by: Bill Little <[email protected]>

* more thorough in_place check

Co-authored-by: Bill Little <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* review actions

* whatsnew

Co-authored-by: Bill Little <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* set default 'unknown' version (#4892)

* Prevent color bar from being misplaced (#4894)

* Prevent color bar from being misplaced

Currently, if `plt.gca()` gives a different Axes object than `axes` in the function `_label`, the color bar will be malpositioned, ending up either on another subfigure, or in another figure window altogether. Specifying the `ax` argument when calling `plt.colorbar` prevents this.

* add whatsnew and test coverage

* check axes and figure

* tidy and add extra axes2 test

Co-authored-by: Bill Little <[email protected]>

* Lockfile update and skip questionable test (#4901)

* Updated environment lockfiles

* skip questionable test

Co-authored-by: Lockfile bot <[email protected]>

* Adapt to the latest setuptools editable configuration. (#4903)

* Adapt to the latest setuptools editable configuration.

* What's new entry.

* What's new typo.

* Update setuptools in Conda requirements.

* pytest gallery tests (#4792)

* pytest parametrize gallery tests

* two context managers as fixtures

* use pathlib instead of os

* docstrings

* use pytest.mark.filterwarnings

* show_replaced_by_check_graphic as fixture

* Revert previous (more readable without)

This reverts commit bdca27cd301fff9de034f14db9da991dac43facf.

* gallery_dir function to variable

* add setup/teardown fixture

* tweak docstring

* remove gallerytests from Makefile

* update dev guide

* doc tweaks

* use monkeypatch to simplify

* move stuff about

* monkeypatch for sys.path; comments and param ids

* revert assert False

* revert deliberate fails

* import_patches fixture

* post rebase check_graphic fix

* update test references

* whatsnew

* test data version

* Updated environment lockfiles (#4904)

Co-authored-by: Lockfile bot <[email protected]>

* Revert Cell.__hash__ fix (#4874)

* Revert Cell.__hash__ fix

* add whatsnew entry

* update comments

* Revert #4701 cell nan equality

Co-authored-by: lbdreyer <[email protected]>

* Add whats new entry for 4734 (#4908)

* Extend new_axis to reshape _DimensionalMetadata objects (#4896)

* extend iris.util.new_axis

* add tests

* fix tests

* Rewrite new_axis; extra stock cube; cube _dimensional_metadata utility and tests

* Add new_axis tests

* Convert to pytest

* Rename broadcast -> expand; review actions

* Add whatsnew entry

Co-authored-by: stephen.worsley <[email protected]>

* Standard names table update (v79) (#4910)

* update cf standard name table

* add whatsnew

* fix whatsnew

* Pin shapely (#4911)

* pin shapely

* update whatsnew

* Improvements to Pandas-to-Iris bridge (#4890)

* First pass using merge to parse ndim Pandas DataFrame.

* Second pass actually doing array manipulation this time.

* Tidying for public consumption.

* Improved index handling.

* Pytest conversion.

* Minor re-factoring.

* Support for copy argument.

* Calendar handling.

* Correct coord reshaping.

* Correct array indexing.

* DataFrame Series agnosticism.

* Docstrings and structure.

* Error as_cube if FUTURE behaviour is set.

* Tests WIP.

* More tests.

* Completed tests.

* Suppress deprecation warning in TestSeriesAsCube.

* Docstring corrections.

* Another docstring fix.

* Another docstring fix.

* Another docstring fix.

* Doctests.

* Doctests.

* Strict expectation of one Cube in tests.

* format_dimensional_metadata typo.

* Rename pandas_array to pandas_structure.

* Leaner warnings for Series with column arguments.

* Fix date tests.

* Back out pandas_ndim FUTURE flag.

* Uncomment code, whoops.

* What's new entry.

* Remove erroneous pass.

* Doctest typo.

* Handle an empty Pandas structure.

* Test showing that dimension order is preserved.

* Informative error for ragged indexes.

* Include shapely pin in setup.cfg (#4917)

* Include shapely pin in setup.cfg.

* What's new entry.

* update whatsnew (#4914)

* Avoid using mutable default. (#4918)

* Fix some @trexfeathers What's New entries (#4920)

* Minor What's New fixes.

* Missing space.

* Update whatsnew for v3.3.0rc0 (#4921)

* update whatsnew

* fix index

* remove template

* address review comments

* Updated environment lockfiles (#4923)

Co-authored-by: Lockfile bot <[email protected]>

* Update whatsnew (#4924)

* reset whatsnew

* remove bugfix section

* address review comments

* Bump peter-evans/create-pull-request from 4.0.4 to 4.1.1 (#4922)

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 4.0.4 to 4.1.1.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](https://github.com/peter-evans/create-pull-request/compare/923ad837f191474af6b1721408744feb989a4c27...18f90432bedd2afd6a825469ffd38aa24712a91d)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Tilde's and wildcard recognition within iris.save. (#4913)

* Draft: Tilde's and wildcard can now be read by iris.save. Modifications and tests provisionally done, docstrings still need editing, with examples and doctests needing being added/editing.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Docstrings updated for expand_filespecs and save

* docstrings for save and expand_filespecs updated to numpy formatting

* Reviewed: Added expansion and relative path tests to expand_filespecs. Ensured list indexing returns single item lists. Minor comment changes.

* Updated latest What's New, and added ESadek-MO to common links

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed isort conflicts

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Unpin shapely (#4925)

* unpin shapely

* update whatsnew

* SUM, COUNT, PROPORTION: don't use dask on numpy arrays (#4905)

* COUNT

* SUM

* tidy up

* increase dask min pin

* fix returned weights when masked

* typo

* docstrings

* whatsnew

* include reviewer in whatsnew

* Remove `setUpClass` from Iris tests. (#4927)

* remove IrisTest.setUpClass

* whatsnew

* Better handle `setuptools_scm` versioning (#4926)

* Use setuptools_scm release-branch-semver version_scheme.

* Properly handle all possible scm-rtd combinations.

* What's new entry.

* Use git stash to protect setuptools_scm from RTD.

* Better version badges.

* Better use of git stash.

* Substitute commit SHAs in place of versions where appropriate.

* Make rtd_version safe for use in shields.io badges.

* Make rtd_version safe for use in shields.io badges.

* Allowed collapsing coordinates with `nbounds != (0, 2)` (#4870)

* Allowed collapsing over coordinates with nbounds!=0,2

* Simplified test

* Added further tests for other warnings

* pre-commit

* Used unittest's assertWarnsRegex

* Added What's new? entry

* fix whatsnew (#4932)

* Updated environment lockfiles (#4936)

Co-authored-by: Lockfile bot <[email protected]>

* update whatsnew title for v3.3 release (#4939)

* Added zonal mean gallery example, continued from @TTV-Intrepid (#4935)

* Added zonal mean gallery example, continued from @TTV-Intrepid

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added TTVIntrepid to common links for What's New functionality

* Fixed zonal mean plotting, add axh

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor visual changes, ax3 plotting fix, removed warning suppression.

* updated imagerepo for zonal means

* updated ci-tests yml to latest iris test data version

* updated what's new

* updated announcements to one line

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#4942)

updates:
- [github.com/psf/black: 22.6.0 → 22.8.0](https://github.com/psf/black/compare/22.6.0...22.8.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove deprecated 'python setup.py test' support (#4948)

* Remove deprecated 'python setup.py test' support

* add whatsnew entry

* purge unused setup clean_source custom command (#4949)

* purge unnecessary setup.py copy_copyright function (#4950)

* fix setup.py std_names building (#4952)

* fix setup.py std_names building

* refactor

* docs whatsnew entry updates (#4953)

* Fix cube repr for scalar anc_var and cell measures (#4945)

* Fix cube repr for scalar anc_var and cell measures

* Add whatsnew

* Fix cube summary tests

* Review actions: reword whats new, reorder scalar sections in cube printout

* Fix cube summary test, update whatsnew

* Fix code comment

* update pypi trove classifiers (#4954)

* Add new Q+A issue, update author email

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Bill Little <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Lockfile bot <[email protected]>
Co-authored-by: Martin Yeo <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bill Little <[email protected]>
Co-authored-by: Ruth Comer <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Patrick Peglar <[email protected]>
Co-authored-by: Will Benfold <[email protected]>
Co-authored-by: Denis Sergeev <[email protected]>
Co-authored-by: tkknight <[email protected]>
Co-authored-by: Will Benfold <[email protected]>
Co-authored-by: Manuel Schlund <[email protected]>
Co-authored-by: stephenworsley <[email protected]>
Co-authored-by: Ruth Comer <[email protected]>
Co-authored-by: iris-actions[bot] <107775138+iris-actions[bot]@users.noreply.github.com>
Co-authored-by: scitools-ci[bot] <107775138+scitools-ci[bot]@users.noreply.github.com>
Co-authored-by: Kristofer Krus <[email protected]>
Co-authored-by: stephen.worsley <[email protected]>
Co-authored-by: Elias <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@trexfeathers trexfeathers trexfeathers approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pp-mo @rcomer @trexfeathers