[DNM] Scatter shuffle proof-of-concept #5473

Fixes dask#4959 `get_client` was calling the private `Worker._get_client` method when it ran within a task. `_get_client` should really have been called `_make_client`, since it created a new client every time. The simplest correct thing to do instead would have been to use the `Worker.client` property, which caches this instance. In order to pass the `timeout` parameter through though, I changed `Worker.get_client` to actually match its docstring and always return the same instance.

If you accidentally pass Futures created by a different Client into `Client.gather`, you'll get a `CancelledError`. This is confusing and misleading. An explicit check for this would have made discovering dask#5466 much easier. And since there are probably plenty of other race conditions regarding default clients in multiple threads, hopefully a better error message will save someone else time in the future too.

This is probably a good idea in general (xref dask#4959), but it particularly helps with performance deserializing Futures, which have a fastpath through `Client.current` that bypasses a number of unnecessarily slow things that `get_client` does before it checks `Client.current`.

When a key is scattered from a task and written directly to worker storage, the Client immediately sets the Future's state to `"finished"`. There's no need for the scheduler to also tell the client that that key is finished; it already knows. This saves a bit of scheduler time and a comms roundtrip.

Helps with setting the current client in worker while deserializing. Implementation referenced from python/cpython#9688

This was really slow and probably doesn't matter when the future is coming from a worker. But probably not safe to remove in general?

We don't need to report back to the client that its key was cancelled. But this shouldn't be exposed and may be wrong.

Just want to see how it affects performance

Hoping this speeds up the transfer of Futures; makes no sense in general though.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DNM] Scatter shuffle proof-of-concept #5473

[DNM] Scatter shuffle proof-of-concept #5473

Commits on Oct 27, 2021