Incompatibility with zarr data created by xarray

[malmans2]

I'm looking into the compatibility between xarray/zarr and netcdf-c.
See also: pydata/xarray#6374, zarr-developers/zarr-specs#41

I'm able to open with xarray zarr data created using netcdf-c + mode=xarray.
However, I'm getting wrong values when I try to access with netcdf-c zarr data created by xarray.

For example, when I run ncdump on zarr data created by xarray, I get this:

>> nc-config --version
netCDF 4.8.1

>> ncdump file://test.xr#mode=xarray
netcdf test {
    dimensions:
    	y = 2 ;
    	x = 3 ;
    variables:
    	double a(y, x) ;
    	double b(y, x) ;
    	int64 x(x) ;
    	int64 y(y) ;
    data:

     a =
      1.01923761795664e-312, 1.35807730645493e-312, 0.147080684863132,
      0.490867372707214, 0.503490103833213, 0.439996145495149 ;

     b =
      1.01923761795664e-312, 1.35807730645493e-312, 0.0463766559811987,
      0.953815138263337, 0.614742013012656, 0.126003533584756 ;

     x = 103216775426, 171798691864, 10 ;

     y = 68857037058, 137438953488 ;
    }

This is the expected output:

>> ncdump file://test.nc#mode=xarray
netcdf test {
    dimensions:
    	y = 2 ;
    	x = 3 ;
    variables:
    	double a(y, x) ;
    		a:_FillValue = NaN ;
    	double b(y, x) ;
    		b:_FillValue = NaN ;
    	int64 x(x) ;
    	int64 y(y) ;
    data:

     a =
      0.147080684863132, 0.490867372707214, 0.503490103833213,
      0.439996145495149, 0.75735182550828, 0.162422659447904 ;

     b =
      0.0463766559811987, 0.953815138263337, 0.614742013012656,
      0.126003533584756, 0.135967994326768, 0.394047363487918 ;

     x = 10, 20, 30 ;

     y = 0, 1 ;
    }

Here is the python code to reproduce the example above.

import xarray as xr
import numpy as np
import zarr
import netCDF4
print("xarray:", xr.__version__)
print("zarr:", zarr.__version__)
print("netCDF4:", netCDF4.__version__)

xarray: 2022.3.0
zarr: 2.11.1
netCDF4: 1.5.8

ds = xr.Dataset(
    {
        "a": (("y", "x"), np.random.rand(6).reshape(2, 3)),
        "b": (("y", "x"), np.random.rand(6).reshape(2, 3)),
    },
    coords={"y": [0, 1], "x": [10, 20, 30]},
)
ds.to_zarr("test.xr")
ds.to_netcdf("file://test.nc#mode=xarray")

kwargs = {}
for ext in ("nc", "xr"):
    for kwargs["engine"] in ("zarr", "netcdf4"):
        path = f"test.{ext}"
        kwargs["filename_or_obj"] = path if kwargs["engine"] == "zarr" else f"file://{path}#mode=xarray"
        print(kwargs, end=": ")
        xr.testing.assert_equal(ds, xr.open_dataset(**kwargs))
        print("OK")

{'engine': 'zarr', 'filename_or_obj': 'test.nc'}: OK
{'engine': 'netcdf4', 'filename_or_obj': 'file://test.nc#mode=xarray'}: OK
{'engine': 'zarr', 'filename_or_obj': 'test.xr'}: OK
{'engine': 'netcdf4', 'filename_or_obj': 'file://test.xr#mode=xarray'}:

AssertionError: Left and right Dataset objects are not equal

Differing coordinates:
L * y        (y) int64 0 1
R * y        (y) int64 68857037058 137438953488
L * x        (x) int64 10 20 30
R * x        (x) int64 103216775426 171798691864 10
Differing data variables:
L   b        (y, x) float64 0.04638 0.9538 0.6147 0.126 0.136 0.394
R   b        (y, x) float64 ...
L   a        (y, x) float64 0.1471 0.4909 0.5035 0.44 0.7574 0.1624
R   a        (y, x) float64 1.019e-312 1.358e-312 0.6229 0.5437 0.6967 0.04557

Am I doing something wrong, or netcdf-c is not compatible with zarr data created using xarray?

cc: @alexamici @aurghs @joshmoore

[WardF]

Tagging @DennisHeimbigner as well.

[DennisHeimbigner]

I'm looking into the compatibility between xarray/zarr and netcdf-c.
First, thank you for doing this; we need all the eyes we can get.

Is there anyway you can send me the xarray zarr data. If it is in a file hierarchy,
you could send me a tar or zip of the file hierarchy and contents. If it is already
in zip format, you could send that.

[dopplershift]

You should be able to attach a zip file to this issue.

[malmans2]

• This is the problematic dataset, created using xarray's backend "zarr": test.xr.zip
• This is the "control" dataset, created using xarray's backend "netcdf4": test.nc.zip

[DennisHeimbigner]

It appears to work with the current netcdf-c main branch. So I guess there was
some error in implementing the blosc filter in the netcdf-c version you are using.

[malmans2]

Good news - Thanks!