From ee0fae00ad0294dedca6a78ed05b97ef0ddcc211 Mon Sep 17 00:00:00 2001
From: Chen Chao <crazyjvm@gmail.com>
Date: Fri, 11 Apr 2014 14:54:58 +0800
Subject: [PATCH] update spark.default.parallelism

actually, the value 8 is only valid in mesos fine-grained mode :
<code>
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
    conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
</code>
---
 docs/configuration.md | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index f3bfd036f4164..c46dfbe5fabfb 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -96,7 +96,11 @@ Apart from these, the following properties are also available, and may be useful
 <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
 <tr>
   <td>spark.default.parallelism</td>
-  <td>8</td>
+  <td>
+    · Mesos fine grained mode: 8<br/>
+    · Local mode: core number of the local machine<br/>
+    · Others: total core number of all executor nodes or 2, whichever is larger
+  </td>
   <td>
     Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
     <code>reduceByKey</code>, etc) when not set by user.