Skip to content

Commit bff05aa

Browse files
holdenkAndrew Or
authored andcommitted
[SPARK-10469] [DOC] Try and document the three options
From JIRA: Add documentation for tungsten-sort. From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" implemented in https://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its corresponding description in http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty there are only 'sort' and 'hash' two options)." Author: Holden Karau <[email protected]> Closes #8638 from holdenk/SPARK-10469-document-tungsten-sort. (cherry picked from commit a76bde9) Signed-off-by: Andrew Or <[email protected]>
1 parent bc70043 commit bff05aa

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

docs/configuration.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -458,9 +458,12 @@ Apart from these, the following properties are also available, and may be useful
458458
<td><code>spark.shuffle.manager</code></td>
459459
<td>sort</td>
460460
<td>
461-
Implementation to use for shuffling data. There are two implementations available:
462-
<code>sort</code> and <code>hash</code>. Sort-based shuffle is more memory-efficient and is
463-
the default option starting in 1.2.
461+
Implementation to use for shuffling data. There are three implementations available:
462+
<code>sort</code>, <code>hash</code> and the new (1.5+) <code>tungsten-sort</code>.
463+
Sort-based shuffle is more memory-efficient and is the default option starting in 1.2.
464+
Tungsten-sort is similar to the sort based shuffle, with a direct binary cache-friendly
465+
implementation with a fall back to regular sort based shuffle if its requirements are not
466+
met.
464467
</td>
465468
</tr>
466469
<tr>

0 commit comments

Comments
 (0)