Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow limiting number of workers running a single job #40

Open
iravanchi opened this issue Sep 10, 2018 · 0 comments
Open

Allow limiting number of workers running a single job #40

iravanchi opened this issue Sep 10, 2018 · 0 comments
Labels
enhancement New feature or request epic

Comments

@iravanchi
Copy link

In job-intensive systems where many background jobs need to be processed in parallel, there can be tens or hundreds of worker nodes in a cluster.

Currently, each worker starts a JobProcessor for ALL of the InProgress jobs. This means all of the nodes will start processing many jobs, which is not efficient. Typically, each single job needs to be run on a few of the cluster nodes, and the jobs can be distributed on different nodes.

Nebula needs to be able to specify the maximum number of nodes processing each single job.

For example, you can deploy a 100-node worker cluster, with 200 running jobs. Current version of Nebula will start 20000 runners across the cluster. But we need to specify, for example, 5 maximum runners per job (set separately on each individual job), so that a total of 1000 runners will be started and distributed on the cluster nodes (each node running an average of 10 runners).

A nice feature can be intelligent distribution of the runners, so when new jobs are started, it will be assigned to the nodes with less traffic/runners.

@iravanchi iravanchi added enhancement New feature or request epic labels Sep 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request epic
Projects
None yet
Development

No branches or pull requests

1 participant