The future of chunk_store in Zarr

[jhamman]

History

Zarr-Python 2 included the concept of a separate chunk_store. This was added very early on (#37) in Zarr's development and has not been widely used. Its biggest flaw is that it effectively is a runtime configuration for the storage layout of a Zarr hierarchy.

Splitting the storage behavior (or location) of metadata and chunks is not a bad idea. In fact, a number of systems have already implemented this (but without using Zarr's chunk_store argument). Examples include:

• Fsspec's Reference FileSystem / Kerchunk
• Earthmover's Arraylake and Icechunk

There are surely more. The point is that each of these have formalized the chunk store concept via a single store and with well defined specification of the storage layout.

Deprecation ahead of 3.0

This issue tracks the deprecation of this feature ahead of the 3.0 release. A deprecation warning will be added to Zarr's 2.18.4 release. Passing chunk_store to constructors and utility functions will raise a NotImplementedError in 3.0.

Formalizing the chunk store concept in 3.0

Zarr Python 3.0 will remove support for a separate chunk store in the API. However, creating custom stores will be supported. Beyond tracking the deprecation of the chunk_store argument in 2.18, this issue will serve as the place to discuss a fresh design for split metadata/chunks stores in Zarr-Python 3.

[d-v-b]

one point in favor of separating chunk storage from metadata / attributes storage is that we expect chunk data to be processed on different types of hardware like GPUs. By contrast I doubt we will need GPUs for processing array metadata / attributes.

In zarr-python today there's no fixed relationship between stores and memory locations, but if there was, then it might make sense to associate arrays with two stores, one using CPU memory (for metadata / attributes) and the store using memory optimized for chunk stuff.