-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/support multiple lams to the Cutout class #113
Merged
b8raoult
merged 15 commits into
ecmwf:develop
from
paulina-t:feature/support-multiple-lams
Nov 8, 2024
Merged
Feature/support multiple lams to the Cutout class #113
b8raoult
merged 15 commits into
ecmwf:develop
from
paulina-t:feature/support-multiple-lams
Nov 8, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
missing copyrigths (ecmwf#111)
for more information, see https://pre-commit.ci
b8raoult
approved these changes
Nov 8, 2024
floriankrb
added a commit
that referenced
this pull request
Nov 15, 2024
* Add contributors (#105) * Add contributors Co-authored-by: Mario Santa Cruz <[email protected]> * Feature/masks (#104) * add masks Co-authored-by: Florian Pinault <[email protected]> * Feature/new checkpoints (#106) * add masks * Feature/new datasets (#99) * main changes * bugfix * few bugs and add unit tests * work with more planetary computer ds * add optional dependencies * qa * make test optional when adls is not installed (#110) * make test optional when adls is not installed * changelog * tests * tests * split tests * Xarray-zarr example dataset recipe (#108) * add a working xarray-zarr example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: b8raoult <[email protected]> * missing copyrigths * missing copyrigths (#111) * missing copyrigths * fixing --test (changing only the behaviour of creating datasets with --test) * more on testing * fix tests * Feature/support multiple lams to the Cutout class (#113) * Enhance Cutout class to support multiple LAMs with hierarchical masking. --------- Co-authored-by: Paulina Met. <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#112) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/psf/black-pre-commit-mirror: 24.8.0 → 24.10.0](psf/black-pre-commit-mirror@24.8.0...24.10.0) - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2) - [github.com/tox-dev/pyproject-fmt: 2.2.4 → v2.5.0](tox-dev/pyproject-fmt@2.2.4...v2.5.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update .pre-commit-config.yaml (#120) * fix qa for the notebook on multilam (#123) * feature/fix qa (#124) * fix qa for the notebook on multilam * fix qa for the yaml for multilam * Naming guidelines not available to all users. fixing #119 (#125) * upload with ssh (#94) * add copy to ssh target * Feature/new checkpoints (#107) * add masks * save masks to checkpoint * name supporting_arrays * better support for cutout * force np.datetime64 is seconds --------- Co-authored-by: Florian Pinault <[email protected]> * Feature/merge (#126) * save masks to checkpoint * force np.datetime64 is seconds * Call filters from anemoi-transform * when merging datasets, consider missing dates * add gcd for frequency --------- Co-authored-by: Florian Pinault <[email protected]> * Feature/use anemoi transform (#127) * Call filters from anemoi-transform --------- Co-authored-by: Florian Pinault <[email protected]> * Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)" * redo "Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)" * fix merge * Update to documentation on using missing dates (#128) * Updated docs on using datasets with missing dates * Simplify ci: run on develop, and on sundays. And disable downstream-ci-hpc. And test only python 3.11. And test only once when PR are updated. And shortest name to read the full description on github.ci. And test with only ubuntu. (same change as for anemoi-utils ecmwf/anemoi-utils#42) (#129) * skipping long tests (#132) --------- Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: b8raoult <[email protected]> Co-authored-by: Mariah Pope <[email protected]> Co-authored-by: Timothy Smith <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Baudouin Raoult <[email protected]> Co-authored-by: paulina-t <[email protected]> Co-authored-by: Paulina Met. <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Håvard Homleid Haugen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds functionality to the
Cutout
class to support multiple Limited Area Models (LAMs), along with a global dataset. This enhancement allows for hierarchical cutouts across multiple LAMs, handling of overlapping regions and different resolutions.Motivation
The primary motivation for this change is to allow the
anemoi-datasets
package to perform cutouts for multiple LAM datasets (such as MEPS and AROME-Arctic) simultaneously, increasing flexibility in data handling.Key Changes
Multi-LAM support in cutout: Enhanced the
Cutout
class to support multiple LAM datasets, handling them hierarchically where overlapping areas are masked sequentially.Grid handling and resolution support: Updated the grids to compute and return grid point counts for each LAM and the global dataset, supporting varying resolutions.
Hierarchical masking: Introduced hierarchical masking of latitude and longitude in overlapping regions for multiple LAMs. The LAMs are provided in a list, with the first element having the highest priority (i.e., it is not masked). LAM no. 2 contains all the points not in LAM no. 1, and so forth. The last LAM in the list has the points that are not in any other LAM, and the global dataset is masked by all the LAMs.
Testing
Notebook Testing: Created a new notebook,
grids_multilam.ipynb
, based ongrids.ipynb
, to plot and visually verify that theCutout
class handles multiple LAM datasets correctly. This includes testing that the cutout works for:- LAMs with different resolutions
- LAMs that do not overlap
- LAMs entirely contained within the domain of another LAM
- LAMs with a coarser resolution than the global dataset
- Compatibility with thinning and cropping
Note that the notebook relies on datasets generated from the files
grids*.yaml
. I added these files for reproducibility, but you might not want to include them in the repository. Let me know if there are other datasets you prefer to use.Dependencies
No new dependencies: This update builds on existing functionality in anemoi-datasets without introducing new dependencies.
Compatibility: Fully compatible with single LAMs as well.
Notes for Reviewers
Potential areas of impact:
- Due to the significant updates in grid handling and masking logic, reviewers may want to verify compatibility with other classes or functions that rely on
Cutout
.- The behavior of the method
collect_supporting_arrays
may need review. I modified it to store all LAM masks and the global mask, but please let me know if this differs from the desired output.Documentation: Please let me know if additional documentation is needed to explain the multi-LAM functionality or if a style change is needed.