Skip to content

Commit

Permalink
update spark.default.parallelism
Browse files Browse the repository at this point in the history
actually, the value 8 is only valid in mesos fine-grained mode :
<code>
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code> 
override def defaultParallelism(): Int = {
    conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
</code>
  • Loading branch information
CrazyJvm committed Apr 11, 2014
1 parent 44f654e commit ee0fae0
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,11 @@ Apart from these, the following properties are also available, and may be useful
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td>spark.default.parallelism</td>
<td>8</td>
<td>
· Mesos fine grained mode: 8<br/>
· Local mode: core number of the local machine<br/>
· Others: total core number of all executor nodes or 2, whichever is larger
</td>
<td>
Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
<code>reduceByKey</code>, etc) when not set by user.
Expand Down

0 comments on commit ee0fae0

Please sign in to comment.