Skip to content

Commit 6154bf4

Browse files
author
Marcelo Vanzin
committed
Docs.
1 parent 8f252da commit 6154bf4

File tree

1 file changed

+20
-0
lines changed

1 file changed

+20
-0
lines changed

docs/configuration.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2114,6 +2114,26 @@ Apart from these, the following properties are also available, and may be useful
21142114
<a href="job-scheduling.html#resource-allocation-policy">description</a>.
21152115
</td>
21162116
</tr>
2117+
<tr>
2118+
<td><code>spark.dynamicAllocation.shuffleTracking.enabled</code></td>
2119+
<td><code>false</code></td>
2120+
<td>
2121+
Experimental. Enables shuffle file tracking for executors, which allows dynamic allocation
2122+
without the need for an external shuffle service. This option will try to keep alive executors
2123+
that are storing shuffle data for active jobs.
2124+
</td>
2125+
</tr>
2126+
<tr>
2127+
<td><code>spark.dynamicAllocation.shuffleTimeout</code></td>
2128+
<td><code>infinity</code></td>
2129+
<td>
2130+
When shuffle tracking is enabled, controls the timeout for executors that are holding shuffle
2131+
data. The default value means that Spark will rely on the shuffles being garbage collected to be
2132+
able to release executors. If for some reason garbage collection is not cleaning up shuffles
2133+
quickly enough, this option can be used to control when to time out executors even when they are
2134+
storing shuffle data.
2135+
</td>
2136+
</tr>
21172137
</table>
21182138

21192139
### Thread Configurations

0 commit comments

Comments
 (0)