Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Scheduler to class #27

Merged
merged 12 commits into from
Nov 28, 2015
Merged

Refactor Scheduler to class #27

merged 12 commits into from
Nov 28, 2015

Conversation

mrocklin
Copy link
Member

Previously the scheduler was a set of coroutines and queues that should be run together. This was successfully done in the Executor class but rarely run separately.

Now we package up the coroutines, queues, and various state collections (dask, dependents, etc.) into a single Scheduler class. We do this with an eye to run a scheduler on a separate machine from where the executor runs. This is the first step in a few large features, notably running a client executor on a local laptop while having the scheduler on the cluster and also having multiple executors connected to the same scheduler.

The Scheduler and Executor connect by two queues (incoming and outgoing). At the moment these queues are connected directly but in the future we might interpose a socket stream.

mrocklin added a commit that referenced this pull request Nov 28, 2015
Refactor Scheduler to class
@mrocklin mrocklin merged commit fad4ce3 into dask:master Nov 28, 2015
@mrocklin mrocklin deleted the scheduler branch November 28, 2015 03:45
@mrocklin mrocklin restored the scheduler branch November 28, 2015 05:20
@mrocklin mrocklin deleted the scheduler branch November 28, 2015 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant