Support GZip/Zlib compression and Shuffle filter for S3 #136
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds support for netCDF files with Gzip/Zlib compression and/or the byte shuffle filter when using the S3 storage backend. The compression and filter integration tests now run when testing against the S3 storage backend. Closes #120
Codecov ReportPatch coverage is
📢 Thoughts on this report? Let us know!. |
|
Please wait for #135 to merge before merging this PR. |
valeriupredoi
approved these changes
Sep 14, 2023
There was a problem hiding this comment.
ah great stuff, man, cheers muchly @markgoddard - one comment, and nothing else 🍺
| if compression: | ||
| # Currently the numcodecs codec IDs map to the supported compression | ||
| # algorithm names in the active storage server. If that changes we may | ||
| # need to maintain a mapping here. |
There was a problem hiding this comment.
we need to pull out the info in this docstring somewhere where it's a bit more visible too (README doubles as docs for now, and plop a link to the table)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for netCDF files with Gzip/Zlib compression and/or the byte
shuffle filter when using the S3 storage backend.
The compression and filter integration tests now run when testing
against the S3 storage backend.
Closes #120