Experimental chunk manifest support

[LDeakin]

Work in progress to enable stores that reference bytes created outside Zarr (via virtualizarr)

• Manifest storage transformer zarr-developers/zarr-specs#287
• Zarr as a "universal reader" for netCDF etc., via new CF decoding codecs zarr-developers/zarr-specs#303

Notes

1. python3 virtualizarr_gen.py
2. Apply corrections for non-conformant zarr.json metadata
3. cargo test chunk_manifest

• Chunk manifests with URIs would make this much more complicated
  • URI parsing and store registration
  • Store caching
  • Auth?
  • ...
• Should absolute paths / relative paths outside of the store be permitted?

TODO

virtualizarr

manifest.json

• virtualizarr should write paths relative to the hierarchy root. Currently they are relative to the working directory.
  • Or is there a more general way of handling this (via metadata changes)? The store must be opened at the same location as the CWD when virtualize.to_zarr was called. Can't just open a store pointing directly at the hierarchy and then open an array at the root store prefix...
• Absolute paths Paths as URIs zarr-developers/VirtualiZarr#243

zarr.json

• Add variable length chunks support

  Cannot concatenate arrays with inconsistent chunk shapes: (...) vs (...) .Requires ZEP003 (Variable-length Chunks)

  • zarrs already supports variable length chunks
• chunk key encoding should match the path encoding ("0.0" etc).
  • the chunk-manifest-json storage transformer should not need to be aware of the chunk key encoding

    "chunk_key_encoding": {
        "configuration": {
            "separator": "."
        },
        "name": "v2"
    },

• Defaults to "codecs": null, without netCDF compression. Change to Conformant ZarrV3 codecs and fill values zarr-developers/VirtualiZarr#193

    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],

• Defaults to "fill_value": null. Must be set to a compatible value. Conformant ZarrV3 codecs and fill values zarr-developers/VirtualiZarr#193

zarrs

• Test on real/multifile netCDF data
• Support NetCDF/HDF5 compression
  • ZLIB codec
  • SZIP codec

[codecov]

Codecov Report

Attention: Patch coverage is 51.68067% with 115 lines in your changes missing coverage. Please review.

Project coverage is 82.75%. Comparing base (46bd880) to head (43a9fa3).
Report is 69 commits behind head on main.

Files with missing lines	Patch %	Lines
...anifest_json/chunk_manifest_storage_transformer.rs	44.11%	114 Missing ⚠️
...c/array/storage_transformer/chunk_manifest_json.rs	95.65%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #79      +/-   ##
==========================================
- Coverage   82.97%   82.75%   -0.23%     
==========================================
  Files         162      166       +4     
  Lines       22500    22738     +238     
==========================================
+ Hits        18670    18816     +146     
- Misses       3830     3922      +92     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[LDeakin]

The manifest storage transformer is unlikely to proceed. Supplanted by https://icechunk.io/spec/