Update to documentation on using missing dates #128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
b8raoult
approved these changes
Nov 14, 2024
floriankrb
added a commit
that referenced
this pull request
Nov 15, 2024
* Add contributors (#105) * Add contributors Co-authored-by: Mario Santa Cruz <[email protected]> * Feature/masks (#104) * add masks Co-authored-by: Florian Pinault <[email protected]> * Feature/new checkpoints (#106) * add masks * Feature/new datasets (#99) * main changes * bugfix * few bugs and add unit tests * work with more planetary computer ds * add optional dependencies * qa * make test optional when adls is not installed (#110) * make test optional when adls is not installed * changelog * tests * tests * split tests * Xarray-zarr example dataset recipe (#108) * add a working xarray-zarr example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: b8raoult <[email protected]> * missing copyrigths * missing copyrigths (#111) * missing copyrigths * fixing --test (changing only the behaviour of creating datasets with --test) * more on testing * fix tests * Feature/support multiple lams to the Cutout class (#113) * Enhance Cutout class to support multiple LAMs with hierarchical masking. --------- Co-authored-by: Paulina Met. <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#112) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/psf/black-pre-commit-mirror: 24.8.0 → 24.10.0](psf/black-pre-commit-mirror@24.8.0...24.10.0) - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.2](astral-sh/ruff-pre-commit@v0.6.9...v0.7.2) - [github.com/tox-dev/pyproject-fmt: 2.2.4 → v2.5.0](tox-dev/pyproject-fmt@2.2.4...v2.5.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update .pre-commit-config.yaml (#120) * fix qa for the notebook on multilam (#123) * feature/fix qa (#124) * fix qa for the notebook on multilam * fix qa for the yaml for multilam * Naming guidelines not available to all users. fixing #119 (#125) * upload with ssh (#94) * add copy to ssh target * Feature/new checkpoints (#107) * add masks * save masks to checkpoint * name supporting_arrays * better support for cutout * force np.datetime64 is seconds --------- Co-authored-by: Florian Pinault <[email protected]> * Feature/merge (#126) * save masks to checkpoint * force np.datetime64 is seconds * Call filters from anemoi-transform * when merging datasets, consider missing dates * add gcd for frequency --------- Co-authored-by: Florian Pinault <[email protected]> * Feature/use anemoi transform (#127) * Call filters from anemoi-transform --------- Co-authored-by: Florian Pinault <[email protected]> * Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)" * redo "Revert "Feature/merge (#126)" "Feature/new checkpoints (#107)" "upload with ssh (#94)" * fix merge * Update to documentation on using missing dates (#128) * Updated docs on using datasets with missing dates * Simplify ci: run on develop, and on sundays. And disable downstream-ci-hpc. And test only python 3.11. And test only once when PR are updated. And shortest name to read the full description on github.ci. And test with only ubuntu. (same change as for anemoi-utils ecmwf/anemoi-utils#42) (#129) * skipping long tests (#132) --------- Co-authored-by: Matthew Chantry <[email protected]> Co-authored-by: b8raoult <[email protected]> Co-authored-by: Mariah Pope <[email protected]> Co-authored-by: Timothy Smith <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Baudouin Raoult <[email protected]> Co-authored-by: paulina-t <[email protected]> Co-authored-by: Paulina Met. <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Håvard Homleid Haugen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Anemoi-training has internal handling that deals with missing dates in the dataset https://github.com/ecmwf/anemoi-training/blob/develop/src/anemoi/training/utils/usable_indices.py, and when training on a dataset with missing dates you do not have to specify this in the dataloader config.
The current documentation on missing dates is confusing because it states that you should specify skip_missing_dates and expected_access which does not work with how missing dates are handled in anemoi-training. I realise that anemoi-datasets can be used for applications outside anemoi-training where the skip_missing_dates functionality can be useful, so I propose adding a short section on how to deal with missing_dates in anemoi-training since this will be the most common use case.
📚 Documentation preview 📚: https://anemoi-datasets--128.org.readthedocs.build/en/128/