Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously the scheduler was a set of coroutines and queues that should be run together. This was successfully done in the
Executor
class but rarely run separately.Now we package up the coroutines, queues, and various state collections (
dask
,dependents
, etc.) into a singleScheduler
class. We do this with an eye to run a scheduler on a separate machine from where the executor runs. This is the first step in a few large features, notably running a client executor on a local laptop while having the scheduler on the cluster and also having multiple executors connected to the same scheduler.The Scheduler and Executor connect by two queues (incoming and outgoing). At the moment these queues are connected directly but in the future we might interpose a socket stream.