Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unstructured scheme lazy regridding (with performance tests) #111

Merged

Conversation

stephenworsley
Copy link
Contributor

Adds lazy regridding to the unstructured schemes equivalent to that added to main in #80. Also added the performance tests for unstructured schemes (including performance tests for lazy regridding) equivalent to those in #98 and #100.
Note that the main difference in code between this and #80 is the need to modify the iris function map_complete_blocks to be able to handle changing the number of dimensions during regridding.

@codecov
Copy link

codecov bot commented Sep 6, 2021

Codecov Report

Merging #111 (9b8df45) into unstructured_scheme (13f8929) will increase coverage by 0.04%.
The diff coverage is 100.00%.

Impacted file tree graph

@@                   Coverage Diff                   @@
##           unstructured_scheme     #111      +/-   ##
=======================================================
+ Coverage                99.43%   99.48%   +0.04%     
=======================================================
  Files                       24       24              
  Lines                     1424     1550     +126     
=======================================================
+ Hits                      1416     1542     +126     
  Misses                       8        8              
Impacted Files Coverage Δ
esmf_regrid/experimental/unstructured_scheme.py 100.00% <100.00%> (ø)
...nstructured_scheme/test_GridToMeshESMFRegridder.py 100.00% <100.00%> (ø)
...nstructured_scheme/test_MeshToGridESMFRegridder.py 100.00% <100.00%> (ø)
...ntal/unstructured_scheme/test__mesh_to_MeshInfo.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13f8929...9b8df45. Read the comment docs.

Copy link
Member

@jamesp jamesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephenworsley looks good, a few comments inline, mostly requests for a little more documentation. Let's catch up on the phone and do a quick overview together too.

Copy link
Contributor

@trexfeathers trexfeathers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @stephenworsley. I have quite a few comments I'm afraid! See what you think.

esmf_regrid/experimental/unstructured_scheme.py Outdated Show resolved Hide resolved
esmf_regrid/experimental/unstructured_scheme.py Outdated Show resolved Hide resolved
esmf_regrid/experimental/unstructured_scheme.py Outdated Show resolved Hide resolved
esmf_regrid/experimental/unstructured_scheme.py Outdated Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Outdated Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Outdated Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Outdated Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Show resolved Hide resolved
Copy link
Contributor

@trexfeathers trexfeathers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes @stephenworsley. Just a couple of minor comments left to address.

As already agreed: please raise an issue on GitHub and on JIRA for restructuring the tests.

We discussed offline yesterday that there was no easy solution to the confusing repetition in benchmarks/ci/esmf_regridder.py. But I very much hope we can revisit this in future, especially once we have a better solution for synthetic data generation. I'm certain that I 90% of the lines I read appear at least twice, in different places, but in slightly different orders - there must surely be a way of generalising some things here if we take a fresh look another time!

benchmarks/benchmarks/ci/esmf_regridder.py Outdated Show resolved Hide resolved
benchmarks/benchmarks/ci/esmf_regridder.py Outdated Show resolved Hide resolved
@jamesp
Copy link
Member

jamesp commented Sep 16, 2021

Great stuff @stephenworsley, @trexfeathers!

@trexfeathers trexfeathers merged commit 9b257cf into SciTools:unstructured_scheme Sep 16, 2021
stephenworsley added a commit to stephenworsley/iris-esmf-regrid that referenced this pull request Jan 10, 2022
* unstructured_scheme: (22 commits)
  Check mesh equality on MeshToGridESMFRegridder call (SciTools#138)
  Formalise regridder file format (SciTools#137)
  Fix issue 135 (mesh to grid chunking problems) (SciTools#136)
  Add load/save benchmarks (SciTools#132)
  Regridder load/saving (SciTools#130)
  Update dependencies (SciTools#128)
  Perform scalability for larger grids (SciTools#122)
  Add performance tests (SciTools#117)
  change iris source (SciTools#115)
  Unstructured scheme lazy regridding (with performance tests) (SciTools#111)
  Updating feature branch unstructured_scheme from 44d6048 to head of main, e528cbf
  Add grid to mesh scheme (SciTools#96)
  Unstructured scheme integration test (SciTools#66)
  Updated lockfiles according to branch requirement spec.
  Update Version to v0.1.dev2 (SciTools#59)
  Update version to v0.1.dev1 (SciTools#58)
  Unstructured Scheme - Extra Dims (SciTools#55)
  add __init__ to tests (SciTools#56)
  Unstructured Scheme - Additional Polish (docstrings and test coverage) (SciTools#53)
  Unstructured Scheme - Cube Creation 3D (SciTools#47)
  ...

# Conflicts:
#	benchmarks/benchmarks/ci/esmf_regridder.py
#	benchmarks/benchmarks/generate_data.py
#	requirements/nox.lock/py36-linux-64.lock
#	requirements/nox.lock/py37-linux-64.lock
#	requirements/nox.lock/py38-linux-64.lock
#	requirements/py36.yml
stephenworsley added a commit to stephenworsley/iris-esmf-regrid that referenced this pull request Jan 31, 2022
* unstructured_scheme: (24 commits)
  reset flag (SciTools#145)
  Use more stable version of connectivity indices (SciTools#144)
  Check mesh equality on MeshToGridESMFRegridder call (SciTools#138)
  Formalise regridder file format (SciTools#137)
  Fix issue 135 (mesh to grid chunking problems) (SciTools#136)
  Add load/save benchmarks (SciTools#132)
  Regridder load/saving (SciTools#130)
  Update dependencies (SciTools#128)
  Perform scalability for larger grids (SciTools#122)
  Add performance tests (SciTools#117)
  change iris source (SciTools#115)
  Unstructured scheme lazy regridding (with performance tests) (SciTools#111)
  Updating feature branch unstructured_scheme from 44d6048 to head of main, e528cbf
  Add grid to mesh scheme (SciTools#96)
  Unstructured scheme integration test (SciTools#66)
  Updated lockfiles according to branch requirement spec.
  Update Version to v0.1.dev2 (SciTools#59)
  Update version to v0.1.dev1 (SciTools#58)
  Unstructured Scheme - Extra Dims (SciTools#55)
  add __init__ to tests (SciTools#56)
  ...

# Conflicts:
#	benchmarks/benchmarks/ci/esmf_regridder.py
#	benchmarks/benchmarks/generate_data.py
#	requirements/nox.lock/py36-linux-64.lock
#	requirements/nox.lock/py37-linux-64.lock
#	requirements/nox.lock/py38-linux-64.lock
#	requirements/py36.yml
#	requirements/py37.yml
@trexfeathers trexfeathers mentioned this pull request Feb 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants