zarr-python v3 compatibility
#516
Conversation
| @@ -496,6 +516,8 @@ def _translator( | |||
| if h5obj.fletcher32: | |||
| logging.info("Discarding fletcher32 checksum") | |||
| v["size"] -= 4 | |||
| key = str.removeprefix(h5obj.name, "/") + "/" + ".".join(map(str, k)) | |||
There was a problem hiding this comment.
This is the same as what _chunk_key did? Maybe make it a function with a comment saying it's a copy/reimplementation.
By the way, is h5obj.name not actually a string, so you could have done h5obj.name.removeprefix()?
kerchunk/hdf.py
Outdated
| shape=h5obj.shape, | ||
| dtype=dt or h5obj.dtype, | ||
| chunks=h5obj.chunks or False, | ||
| fill_value=fill, | ||
| compression=None, | ||
| compressor=None, |
There was a problem hiding this comment.
So here, you could reintroduce the compressor
filters = filters[:-1]
compressor = filters[-1]
but obviously it depends on whether there are indeed any filters at all.
It would still need back compat, since filters-only datasts definitely exist.
There was a problem hiding this comment.
yeah the big issue is that v3 cares about what type of operation it is, and v2w doesnt so moving them around doesnt necessarily fix that bug
There was a problem hiding this comment.
So there needs to be a change upstream?
There was a problem hiding this comment.
Yes this: zarr-developers/zarr-python#2325
kerchunk/hdf.py
Outdated
| for k, v in self.store_dict.items(): | ||
| if isinstance(v, zarr.core.buffer.cpu.Buffer): | ||
| key = str.removeprefix(k, "/") | ||
| new_keys[key] = v.to_bytes() | ||
| keys_to_remove.append(k) | ||
| for k in keys_to_remove: | ||
| del self.store_dict[k] |
There was a problem hiding this comment.
This is the hacky bit and could use some explanations. Even when requesting "v2", zarr makes Buffer objects, and the keys are also wrong?
There was a problem hiding this comment.
Yeah so two issues here:
- the keys we get from hdf are for example
/depth/.zarraywhen then need to bedepth/.zarray - we cant jsonify buffers, which is how the internal
MemoryStorein v3 stores its data. So we need to convert the buffers to bytes to be serialized
There was a problem hiding this comment.
OK - would appreciate comments on the code saying this.
|
Also worth noting... zarr 3 doesn't support numcodecs codecs out of the box. There is a pr to help this zarr-developers/numcodecs#524 (see also here for updated version zarr-developers/numcodecs#597) but it would mean a change to codec names which causes an incompatibility. For the initial icechunk examples we handle this in virtualizarr but long term it probably belongs here to work standalone. |
|
When this zarr-developers/zarr-python#2425 goes in it should unblock this to work full with zarr python v3. We will also need to create both numcodecs and zarr v3 codec versions of all the custom kerchunk codecs so that a given dataset can be loaded in either v2 or v3 contexts (say if you kerchunk a grib file, then want to convert those references to an icechunk store) |
Is that a subclass in each case, specifying that it is to be bytes->bytes? |
|
Sorry the numcodecs versions exists as they are today. Yes the v3 version would be basically subclassing zarr.abc.codec using the numcodecs implementations of the kerchunk codecs. Although grib decoder should really be bytes to array i think |
|
This is required upstream: fsspec/filesystem_spec#1734 |
|
Also of note: To get this to work with zarr 3, we pass an fsspec |
|
Grib works as long as read in zarr 2 format. To be used with zarr 3 the codec needs to be ported over to the new abc.Codec class from zarr |
|
I think one change that should be made for compatibility with zarr 3 and virtualizarr is that the Grib2 codec moves from filters to compressors. |
It returns an array, though - this was already pointed out. I think there is an assumption that compressors map to byte->byte codecs. |
|
Ill change it back for now. Will discuss with zarr folks |
implement zarr3 support for grib
|
State of the Union post merge with @moradology and installing latest fsspec and zarr 3
A lot of these are the same errors over and over a again. The hardest part will be maintaining compat for zarr python 2 library which I am not sure if it should be a goal of this PR @martindurant |
So far the only tested file type is HDF5. Thats the only module that currently works with new zarr python in some way