Enable file locks in MetDataSource.open_dataset

[thabbott]

Changes

These changes prevent segmentation faults when reading and writing multi-file datasets with recent netCDF4 versions (see #204) and allow us to remove upper limits on netCDF4 and numpy versions.

These changes revert a change (#68) made to address the issue described in pydata/xarray#4406. That issue remains open, so this change may resurface problems with dask threadlocks.

Breaking changes

• Remove lock=False as a default keyword argument to xr.open_mfdataset in the MetDataSource.open_dataset method. This reverts a change from v0.44.1 and prevents segmentation faults when using recent versions of netCDF4 (1.7.0 and above).

Internals

• Remove upper limits on netCDF4 and numpy versions.

Tests

• QC test passes locally (make test)
• CI tests pass

Reviewer

@zebengberg

[zebengberg]

Thanks Tristan!

[zebengberg]

So will the doctests now fail with numpy<2?

[thabbott]

Yes. I added a check in the Makefile (d39d70e) to make this requirement explicit.

Trying to keep doctests compatible with numpy 1 and 2 seems tricky.... You can get the same output for scalars by using print, but that doesn't work for scalars wrapped in some other object:

>>> np.float64(1.0)
1.0              # numpy 1
np.float64(1.0)  # numpy 2
>>> print(np.float64(1.0)
1.0  # numpy 1
1.0  # numpy 2
>>> print((np.float64(1.0), np.float64(2.0))
(1.0, 2.0)                          # numpy 1
(np.float64(1.0), np.float64(2.0))  # numpy 2

Happy to discuss this more if you don't think this seems like the right solution!