update spark.default.parallelism #389
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
actually, the value 8 is only valid in mesos fine-grained mode :
<code>
override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>
while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
</code>
|
Can one of the admins verify this patch? |
|
Jenkins, test this please |
|
Good catch |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14084/ |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14086/ |
| <ul> | ||
| <li>Mesos fine grained mode: 8 | ||
| <li>Local mode: core number of the local machine | ||
| <li>Others: total core number of all executor nodes or 2, whichever is larger |
There was a problem hiding this comment.
Actually to have valid HTML, add </li> at the end of these.
oh yeah, missed </li>. fixed it.
CrazyJvm
added a commit
to CrazyJvm/spark
that referenced
this pull request
Apr 14, 2014
"By default, this uses only 8 parallel tasks to do the grouping." is a big misleading. Please refer to apache#389 detail is as following code : <code> def defaultPartitioner(rdd: RDD[_], others: RDD[_]*): Partitioner = { val bySize = (Seq(rdd) ++ others).sortBy(_.partitions.size).reverse for (r <- bySize if r.partitioner.isDefined) { return r.partitioner.get } if (rdd.context.conf.contains("spark.default.parallelism")) { new HashPartitioner(rdd.context.defaultParallelism) } else { new HashPartitioner(bySize.head.partitions.size) } } </code>
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Jenkins test result is OK, but Travis fails. So...what's going on? |
|
Built this locally and it looked gogd, so I'm merging it. Don't worry about Travis - it's currently experimental. |
asfgit
pushed a commit
that referenced
this pull request
Apr 16, 2014
actually, the value 8 is only valid in mesos fine-grained mode :
<code>
override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>
while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
</code>
Author: Chen Chao <[email protected]>
Closes #389 from CrazyJvm/patch-2 and squashes the following commits:
84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism
(cherry picked from commit 9edd887)
Signed-off-by: Patrick Wendell <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Apr 17, 2014
"By default, this uses only 8 parallel tasks to do the grouping." is a big misleading. Please refer to #389 detail is as following code : def defaultPartitioner(rdd: RDD[_], others: RDD[_]*): Partitioner = { val bySize = (Seq(rdd) ++ others).sortBy(_.partitions.size).reverse for (r <- bySize if r.partitioner.isDefined) { return r.partitioner.get } if (rdd.context.conf.contains("spark.default.parallelism")) { new HashPartitioner(rdd.context.defaultParallelism) } else { new HashPartitioner(bySize.head.partitions.size) } } Author: Chen Chao <[email protected]> Closes #403 from CrazyJvm/patch-4 and squashes the following commits: 42f6c9e [Chen Chao] fix format 829a995 [Chen Chao] fix format 1568336 [Chen Chao] misleading task number of groupByKey
asfgit
pushed a commit
that referenced
this pull request
Apr 17, 2014
"By default, this uses only 8 parallel tasks to do the grouping." is a big misleading. Please refer to #389 detail is as following code : def defaultPartitioner(rdd: RDD[_], others: RDD[_]*): Partitioner = { val bySize = (Seq(rdd) ++ others).sortBy(_.partitions.size).reverse for (r <- bySize if r.partitioner.isDefined) { return r.partitioner.get } if (rdd.context.conf.contains("spark.default.parallelism")) { new HashPartitioner(rdd.context.defaultParallelism) } else { new HashPartitioner(bySize.head.partitions.size) } } Author: Chen Chao <[email protected]> Closes #403 from CrazyJvm/patch-4 and squashes the following commits: 42f6c9e [Chen Chao] fix format 829a995 [Chen Chao] fix format 1568336 [Chen Chao] misleading task number of groupByKey (cherry picked from commit 9c40b9e) Signed-off-by: Reynold Xin <[email protected]>
CrazyJvm
added a commit
to CrazyJvm/spark
that referenced
this pull request
May 14, 2014
<code>
private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
</code>
it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
the property "spark.default.parallelism" refers to apache#389
asfgit
pushed a commit
that referenced
this pull request
May 15, 2014
private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
the property "spark.default.parallelism" refers to #389
Author: Chen Chao <[email protected]>
Closes #766 from CrazyJvm/patch-7 and squashes the following commits:
0b7efba [Chen Chao] Update streaming-programming-guide.md
cc5b66c [Chen Chao] default task number misleading in several places
(cherry picked from commit 2f63995)
Signed-off-by: Reynold Xin <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
May 15, 2014
private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
the property "spark.default.parallelism" refers to #389
Author: Chen Chao <[email protected]>
Closes #766 from CrazyJvm/patch-7 and squashes the following commits:
0b7efba [Chen Chao] Update streaming-programming-guide.md
cc5b66c [Chen Chao] default task number misleading in several places
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
actually, the value 8 is only valid in mesos fine-grained mode :
<code>
override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>
while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
</code>
Author: Chen Chao <[email protected]>
Closes apache#389 from CrazyJvm/patch-2 and squashes the following commits:
84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
"By default, this uses only 8 parallel tasks to do the grouping." is a big misleading. Please refer to apache#389 detail is as following code : def defaultPartitioner(rdd: RDD[_], others: RDD[_]*): Partitioner = { val bySize = (Seq(rdd) ++ others).sortBy(_.partitions.size).reverse for (r <- bySize if r.partitioner.isDefined) { return r.partitioner.get } if (rdd.context.conf.contains("spark.default.parallelism")) { new HashPartitioner(rdd.context.defaultParallelism) } else { new HashPartitioner(bySize.head.partitions.size) } } Author: Chen Chao <[email protected]> Closes apache#403 from CrazyJvm/patch-4 and squashes the following commits: 42f6c9e [Chen Chao] fix format 829a995 [Chen Chao] fix format 1568336 [Chen Chao] misleading task number of groupByKey
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
the property "spark.default.parallelism" refers to apache#389
Author: Chen Chao <[email protected]>
Closes apache#766 from CrazyJvm/patch-7 and squashes the following commits:
0b7efba [Chen Chao] Update streaming-programming-guide.md
cc5b66c [Chen Chao] default task number misleading in several places
tangzhankun
pushed a commit
to tangzhankun/spark
that referenced
this pull request
Aug 9, 2017
mccheah
pushed a commit
to mccheah/spark
that referenced
this pull request
Nov 28, 2018
…dle (apache#389) [SPARK-24767] Propagate MDC to spark-submit thread in InProcessAppHandle (apache#389) [SPARK-24813][BUILD][FOLLOW-UP][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Apply AS enabled flavor in FusionCloud job
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
actually, the value 8 is only valid in mesos fine-grained mode :
override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}