-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch to shared Lock (SerializableLock if possible) for reading/writing #1179
Conversation
…writing Fixes pydata#1172 The serializable lock will be useful for dask.distributed or multi-processing (xref pydata#798, pydata#1173, among others).
Is there a clear fail case we can use as a test to demonstrate the value here? |
@mrocklin We could update our dask-distributed integration tests to avoid |
I have found a fail case related to distributed: attempting to use Consider this example: import dask.array as da
from distributed import Client
import xarray as xr
def create_and_store_dataset():
shape = (10000, 1000)
chunks = (1000, 1000)
data = da.zeros(shape, chunks=chunks)
ds = xr.DataArray(data).to_dataset()
ds.to_netcdf('test_dataset.nc')
print("Success!")
create_and_store_dataset()
client = Client()
create_and_store_dataset() The first call succeeds, while the second fails with When using the distributed client, I can successfully call |
Dask 0.13.0 has been released |
I will update the xarray/dask-distributed integration and submit this later today. @rabernat it should solve your issues with |
Hmm. It looks like we need dask 0.13 in conda to make the distributed integration pass. |
It's up now on conda-forge if you're interested in switching over. |
In it goes. We're using |
Fixes #1172
The serializable lock will be useful for dask.distributed or multi-processing
(xref #798, #1173, among others).