Refactor xr.save_mfdataset() to automatically save an xarray object backed by dask arrays to multiple files

[andersy005]

Is your feature request related to a problem? Please describe.

Currently, when a user wants to write multiple netCDF files in parallel with xarray and dask, they can take full advantage of xr.save_mfdataset() function. This function in its current state works fine, but the existing API requires that

• the user generates file paths themselves
• the user maps each chunk or dataset to a corresponding output file

A few months ago, I wrote a blog post showing how to save an xarray dataset backed by dask into multiple netCDF files, and since then I've been meaning to request a new feature to make this process convenient for users.

Describe the solution you'd like

Would it be useful to actually refactor the existing xr.save_mfdataset() to automatically save an xarray object backed by dask arrays to multiple files without needing to create paths ourselves? Today, this can be achieved via xr.map_blocks. In other words, is it possible to have something analogous to to_zarr(....) but for netCDF:

ds.save_mfdataset(prefix="directory/my-dataset")

# or

xr.save_mfdataset(ds, prefix="directoy/my-dataset")

---->

directory/my-dataset-chunk-1.nc
directory/my-dataset-chunk-2.nc
directory/my-dataset-chunk-3.nc
....

[shoyer]

I agree, this does sound useful!

It might make sense to split this into a few pieces of functionality:

1. A new helper function that splits an xarray object into separate objects for each chunk, including some representation of the "chunk id". Perhaps split_chunks()?
2. A new higher level function that combines (1) and the existing save_mfdataset to automatically save an xarray object into multiple files. This probably should be a new function rather than using the existing save_mfdataset because the API is different.

[dcherian]

Perhaps split_chunks()?

There was a proposal for .blocks (#3147 (comment)) I agree that 'chunk id' would be useful. Does dask expose that somehow?