Adding reader_options kwargs to open_virtual_dataset.

[norlandrhagen]

In a step to start reading remote files #61, this PR adds in reader_options to the open_virtual_dataset function.

These reader_options are passed into each Kerchunk file reader (SingleHdf5ToZarr, NetCDF3ToZarr, etc..) in read_kerchunk_references_from_file. Once open_virtual_dataset is replaced with the Xarray backend, we could pass them through: ds = xr.open_dataset(fp, engine='virtualizarr', backend_kwargs={'reader_options': {'storage_options': {'anon': True}}}).

This approach relies on the user knowing what options are available in each Kerchunk file reader.

This example should work off of this PR pointing to a public s3 bucket.

from virtualizarr import open_virtual_dataset
path = 's3://carbonplan-share/virtualizarr/local.nc'

vds = open_virtual_dataset(path)

edit: Tests are passing. Index generation and filetype guessing are now working.

[codecov]

Codecov Report

Attention: Patch coverage is 16.66667% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 74.31%. Comparing base (f226093) to head (4c6cb63).
Report is 20 commits behind head on main.

❗ Current head 4c6cb63 differs from pull request most recent head e6f047f. Consider uploading reports for the commit e6f047f to get more accurate results

Files	Patch %	Lines
virtualizarr/tests/test_xarray.py	25.00%	6 Missing ⚠️
virtualizarr/kerchunk.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #67       +/-   ##
===========================================
- Coverage   90.18%   74.31%   -15.87%     
===========================================
  Files          14       14               
  Lines         998      946       -52     
===========================================
- Hits          900      703      -197     
- Misses         98      243      +145     

Flag	Coverage Δ
unittests	74.31% <16.66%> (-15.87%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TomNicholas]

@norlandrhagen I've merged main in here because @jbusecke is using this branch for testing with CMIP6 data in #93. It would be great to get this PR into main!

[TomNicholas]

Thanks @norlandrhagen ! I think this is a nice minimal addition to support reading from s3. I like how the fsspec stuff is generally kept separate too.

[TomNicholas]

When you normally point xr.open_dataset at an S3 url, you don't need to pass reader_options do you? Can we try to follow the signature of xr.open_dataset as closely as possible? (Maybe this already is as close as we can get)

[norlandrhagen]

I could be wrong, but I thought you had to pass in some sort of fsspec/s3fs mapper.

For me this fails:

ds = xr.open_dataset('s3://carbonplan-share/virtualizarr/local.nc')

[norlandrhagen]

Looks like the docs + CI builds are failing with:

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone --filter=blob:none --quiet https://github.com/TomNicholas/xarray.git /tmp/pip-install-rmq_i38f/xarray_61220b7470e0456fa17857a7af71f04b
  WARNING: Did not find branch or tag 'concat-avoid-index-auto-creation', assuming revision or ref.
  Running command git checkout -q concat-avoid-index-auto-creation
  error: pathspec 'concat-avoid-index-auto-creation' did not match any file(s) known to git
  error: subprocess-exited-with-error

@TomNicholas should we pin another version of Xarray?

[TomNicholas]

should we pin another version of Xarray?

pydata/xarray#8872 was merged yesterday so now we should be able to release xarray, remove the xarray pin in virtualizarr, then release the first version of virtualizarr!

EDIT: Tracking the xarray release pydata/xarray#9018

[TomNicholas]

This seems very close now @norlandrhagen ?

[norlandrhagen]

CI is passing now @TomNicholas!

[TomNicholas]

Thanks @norlandrhagen ! One final request: Can we add a quick explanatory line to the docs? Something like

To open remote files as virtual datasets, pass the reader_kwargs options, e.g.

vds = open_virtual_dataset("s3://fake-bucket/file.nc", reader_kwargs={whatever would be needed})

This would go on the usage page, either as a quick entry underneath the first Opening files as virtual datasets heading, or under a new heading at the bottom.

[TomNicholas]

Thank you so much @norlandrhagen ! Will merge this now

[jbusecke]

Fantastic. Thanks so much. Ill refactor my code in the coming days.