Skip to content

Commit a76bde9

Browse files
holdenkAndrew Or
authored andcommitted
[SPARK-10469] [DOC] Try and document the three options
From JIRA: Add documentation for tungsten-sort. From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" implemented in https://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its corresponding description in http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty there are only 'sort' and 'hash' two options)." Author: Holden Karau <[email protected]> Closes apache#8638 from holdenk/SPARK-10469-document-tungsten-sort.
1 parent e048111 commit a76bde9

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

docs/configuration.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -447,9 +447,12 @@ Apart from these, the following properties are also available, and may be useful
447447
<td><code>spark.shuffle.manager</code></td>
448448
<td>sort</td>
449449
<td>
450-
Implementation to use for shuffling data. There are two implementations available:
451-
<code>sort</code> and <code>hash</code>. Sort-based shuffle is more memory-efficient and is
452-
the default option starting in 1.2.
450+
Implementation to use for shuffling data. There are three implementations available:
451+
<code>sort</code>, <code>hash</code> and the new (1.5+) <code>tungsten-sort</code>.
452+
Sort-based shuffle is more memory-efficient and is the default option starting in 1.2.
453+
Tungsten-sort is similar to the sort based shuffle, with a direct binary cache-friendly
454+
implementation with a fall back to regular sort based shuffle if its requirements are not
455+
met.
453456
</td>
454457
</tr>
455458
<tr>

0 commit comments

Comments
 (0)