Configurable workers #46
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -42,6 +49,10 @@ | |||
| # Make tachyon_mb as spark_mb for now. | |||
| tachyon_mb = spark_mb | |||
|
|
|||
| worker_instances = int(os.getenv("SPARK_WORKER_INSTANCES", 1)) | |||
| # Distribute equally cpu cores among worker instances | |||
| worker_cores = slave_cpus / worker_instances | |||
There was a problem hiding this comment.
It might be good to do:
max(slave_cpus / worker_instances, 1)
I some divergent cases people might try to over-subscribe the machines. It would be good to have Spark at least give 1 core to each worker in these cases.
|
This is cool - just suggested one minor change, but looks good other than that. |
|
Fixed the over-subscribing issue. |
asfgit
pushed a commit
to apache/spark
that referenced
this pull request
May 4, 2014
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: mesos/spark-ec2#46 Author: Allan Douglas R. de Oliveira <[email protected]> Closes #612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter (cherry picked from commit 4669a84) Signed-off-by: Patrick Wendell <[email protected]>
asfgit
pushed a commit
to apache/spark
that referenced
this pull request
May 4, 2014
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: mesos/spark-ec2#46 Author: Allan Douglas R. de Oliveira <[email protected]> Closes #612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
Merged
pwendell
added a commit
that referenced
this pull request
May 4, 2014
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: mesos/spark-ec2#46 Author: Allan Douglas R. de Oliveira <[email protected]> Closes apache#612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added configuration for SPARK_MASTER_OPTS, SPARK_WORKER_INSTANCES and SPARK_WORKER_CORES