[Feature]: S3 streaming support via fsspec

[oruebel]

What would you like to see added to HDMF-ZARR?

Support streaming using fsspec

Is your feature request related to a problem?

NeurodataWithoutBorders/helpdesk#56 (comment)

What solution would you like?

Add FSStore to support fsspect-based streaming from S3 and/or allow passing of a zarr.Group with the parent group as a read-only store for ZarrIO.__init__(path=...)

https://github.com/hdmf-dev/hdmf-zarr/blob/70bf35b60ed2c2eaae7b12080bc1f4cc3d89ba3e/src/hdmf_zarr/backend.py#L66C1-L69C47

Do you have any interest in helping implement the feature?

Yes.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this change was not already requested?

[alejoe91]

@oruebel this came up elsewhere (see comment). Note that zarr natively supports reading from the cloud. On my side, this works

import zarr

remote_zarr_location =  = "s3://aind-open-data/ecephys_625749_2022-08-03_15-15-06_nwb_2023-05-16_16-34-55/ecephys_625749_2022-08-03_15-15-06_nwb/ecephys_625749_2022-08-03_15-15-06_experiment1_recording1.nwb.zarr/"

zarr_root = zarr.open(remote_zarr_location)

When trying with the NWBZarrIO wrapper, some links failed to be resolved because the resolve link function assumes the file/folder is sitting on disk. It would be probably an easy fix to make it work! I can give it a try!