Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with NCZarr. #22

Closed
dblodgett-usgs opened this issue Apr 26, 2023 · 3 comments
Closed

Compatibility with NCZarr. #22

dblodgett-usgs opened this issue Apr 26, 2023 · 3 comments

Comments

@dblodgett-usgs
Copy link

I've just left comments in #18 and #3 asking about this, perhaps you can summarize your thoughts here @christophenoel ?

Scanning this issue: zarr-developers/zarr-specs#41

I tend to think that NCZarr is a lower level concern and that GeoZarr should try to be agnostic to the particular way that NetCDF attributes are encoded into Zarr. That is, I think that GeoZarr should work whether you also need features supported by NCZarr or are using vanilla Zarr. Someone with a deeper knowledge of the landcape will need to chime in here though. @rabernat or someone else -- can you recommend and pull in someone who might be able to help recommend a path forward on this topic?

@dblodgett-usgs
Copy link
Author

From: 21-050r1_Zarr_Community_Standard.pdf

Beginning with NetCDF-C version 4.8.0, Unidata introduced experimental Zarr support
into the NetCDF-C library. This was accomplished via creating a new specification -
NCZarr - which is “similar to, but not identical with the Zarr Version 2 Specification.”
Specifically, NCZarr adds two additional metadata files (“.nczarray" and ".nczattr”),
which are not part of the Zarr V2 Spec. Since NCZarr stores are not fully compatible and
interoperable with Zarr V2, this community standard excludes NCZarr. Work is ongoing
to reconcile NCZarr and the architectural reasons that motivated its development with the
forthcoming Zarr V3 Specification.
Fortunately, the NetCDF-C library also supports reading / writing of data using the
simpler Named Dimension convention described in 4.1.

This seems to indicate that what I stated above, that NCZarr is a lower level concern is basically true.

@christophenoel
Copy link

I don't think this really helps, but I found the following section written by the team in the Cloud-native Format TradeOff report (HDSA project):

NCZarr origins from the need of a pure C implementation of the Zarr API. Starting from 2018, Unidata NetCDF team joined the discussion and provided effort to bring support of Zarr datastores to the existing netcdf-c library. This effort resulted in the creation of an extension of the Zarr specification called NCZarr, which provides a data model close to the existing netcdf-4 data model.
Starting from NetCDF version 4.8.0 (April 2021), the Zarr (and NCZarr) data formats are supported by the netcdf-c library. Since then, the netcdf-c library can provide access to cloud storage such as Amazon S3.
The first NCZarr version relies on special objects that are not part of the Zarr specification (and not supported by the official Zarr library). In the new version of NCZarr, this is no longer the case. The new NCZarr version relies only on pure Zarr objects, using predefined named attributes as a convention. Those specific attributes shall not generate any error when reading Zarr files but might be ignored depending on the Zarr library implementation used.

@dblodgett-usgs
Copy link
Author

That aligns with what's been discussed elsewhere. I'll close this issue as I think we've determined that NCZarr is not an issue for geozarr to overcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants