Skip to content

Commit 9edd887

Browse files
CrazyJvmpwendell
authored andcommitted
update spark.default.parallelism
actually, the value 8 is only valid in mesos fine-grained mode : <code> override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8) </code> while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers! <code> override def defaultParallelism(): Int = { conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2)) } </code> Author: Chen Chao <[email protected]> Closes #389 from CrazyJvm/patch-2 and squashes the following commits: 84a7fe4 [Chen Chao] miss </li> at the end of every single line 04a9796 [Chen Chao] change format ee0fae0 [Chen Chao] update spark.default.parallelism
1 parent fec462c commit 9edd887

File tree

1 file changed

+7
-1
lines changed

1 file changed

+7
-1
lines changed

docs/configuration.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful
9696
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
9797
<tr>
9898
<td>spark.default.parallelism</td>
99-
<td>8</td>
99+
<td>
100+
<ul>
101+
<li>Mesos fine grained mode: 8</li>
102+
<li>Local mode: core number of the local machine</li>
103+
<li>Others: total core number of all executor nodes or 2, whichever is larger</li>
104+
</ul>
105+
</td>
100106
<td>
101107
Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
102108
<code>reduceByKey</code>, etc) when not set by user.

0 commit comments

Comments
 (0)