-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
xarray.open_zarr to be deprecated #70
Comments
zarr-developers/zarr-python#546 will add the get_mapper route into zarr itself, so that you can pass URL and storage_options. This may take a while to merge and release, though. However, I am surprised: I would think that with engine=zarr, xarray would pass through the mapper object to zarr, which knows of course what to do with it. Is the CF decode the only step that xarray is adding? |
Good to know about the zarr development route. I can't wait to see the new behavior, although I understand the timeline... Here is a better description of the problem with store = 'directoryA/subdirB/zarr_store'
_mapper = fsspec.get_mapper(store)
ds = xarray.open_dataset(_mapper, chunks="auto",...) raises the error: ValueError: can only read bytes of file-like objects with engine="scipy" or "h5netcdf" Now, In []: isinstance(_mapper, str)
out []: False
In []: isinstance(_mapper, Path)
Out []: False
In []: isinstance(_mapper, AbstractDatastore)
Out []: False At least one of the above needs to be |
Right - I would think that when engine="zarr", there should be an additional case |
Yes, that does it. That is In []: from collections.abc import MutableMapping
In []: isinstance(mapper, MutableMapping)
Out []: True Thanks @martindurant ! |
With this sorted (and working), should I submit a PR to set |
Presumably yes, after xarray is released |
FYI, pydata/xarray#4187 was merged (a continuation of pydata/xarray#4003) and should be available for the next release of xarray (0.16.2?). Note that Still, it might be preferable to switch |
Right, so I believe |
Awesome, I can take a look in the next couple of days and see what I can do. It shouldn't be much trouble, I hope. Thanks a lot @weiji14! |
In the future, zarr files/stores will be opened with
xarray.open_dataset
as followsThus, (eventually) there needs to be a change on how
intake-xarray
will open zarr stores. Naively, this can be done by specifyingopen_dataset
rather thanopen_zarr
. This is, something like:However,
xarray.open_dataset
does not recognize output fromfsspec.get_mapper
. It works ifstore
(as defined above) is passed.On
xarray.open_zarr
,_mapper
gets transformed into aZarrStore
and later decoded. This is, given the_mapper
, the following will open the zarr store:This brings two options IMHO:
fsspec.get_mapper
(not likely) and just pass the url/path as argument toxarray.open_dataset.
(very unlikely), orxarray.open_dataset
directly, importZarrStore.open_group
andconvenctions.decode_cf
to open zarr stores.It is my understanding, that zarr will potentially depend more on
fsspec
as it gets more developed, and thus No. 2 seems more likely.Or, is there another secret option number 3 I fail to see?
pydata/xarray#4003
fsspec/filesystem_spec#286
zarr-developers/zarr-python#546
The text was updated successfully, but these errors were encountered: