This repository has been archived by the owner on Jan 9, 2020. It is now read-only.
Allow number of executor cores to have fractional values #361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
foxish
reviewed
Jun 29, 2017
| @@ -388,7 +388,8 @@ private[spark] class KubernetesClusterSchedulerBackend( | |||
| val requiredEnv = Seq( | |||
| (ENV_EXECUTOR_PORT, executorPort.toString), | |||
| (ENV_DRIVER_URL, driverUrl), | |||
| (ENV_EXECUTOR_CORES, executorCores), | |||
| // Executor backend expects integral value for executor cores, so round it up to an int. | |||
| (ENV_EXECUTOR_CORES, math.ceil(executorCores.toDouble).toInt.toString), | |||
There was a problem hiding this comment.
We could make it a double when we fetch it in KubernetesClusterSchedulerBackend#L110. That can be replaced by conf.getDouble(..., 1.0)
|
LGTM except for that one comment. |
This commit tries to solve issue #359 by allowing the `spark.executor.cores` configuration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variable `SPARK_EXECUTOR_CORES`, the value is rounded up to the closest integer as required by the `CoarseGrainedExecutorBackend`. Signed-off-by: Yinan Li <[email protected]>
|
Thanks @foxish for reviewing! |
foxish
pushed a commit
that referenced
this pull request
Jul 24, 2017
This commit tries to solve issue #359 by allowing the `spark.executor.cores` configuration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variable `SPARK_EXECUTOR_CORES`, the value is rounded up to the closest integer as required by the `CoarseGrainedExecutorBackend`. Signed-off-by: Yinan Li <[email protected]>
ifilonenko
pushed a commit
to ifilonenko/spark
that referenced
this pull request
Feb 26, 2019
…logging Format bad jars log.
puneetloya
pushed a commit
to puneetloya/spark
that referenced
this pull request
Mar 11, 2019
…k-on-k8s#361) This commit tries to solve issue apache-spark-on-k8s#359 by allowing the `spark.executor.cores` configuration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variable `SPARK_EXECUTOR_CORES`, the value is rounded up to the closest integer as required by the `CoarseGrainedExecutorBackend`. Signed-off-by: Yinan Li <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit tries to solve issue #359 by allowing the
spark.executor.coresconfiguration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variableSPARK_EXECUTOR_CORES, the value is rounded up to the closest integer as expected by theCoarseGrainedExecutorBackend. I expect most use cases of this are to use a fractional of 1 CPU core, e.g., 0.1 or 0.5.Signed-off-by: Yinan Li [email protected]