Skip to content

Commit 28b1af7

Browse files
Marcelo VanzinAndrew Or
authored andcommitted
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.
Author: Marcelo Vanzin <[email protected]> Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits: 554cc07 [Marcelo Vanzin] Change message. 9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
1 parent 53befac commit 28b1af7

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

core/src/main/scala/org/apache/spark/SparkContext.scala

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1396,6 +1396,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
13961396
* Register an RDD to be persisted in memory and/or disk storage
13971397
*/
13981398
private[spark] def persistRDD(rdd: RDD[_]) {
1399+
_executorAllocationManager.foreach { _ =>
1400+
logWarning(
1401+
s"Dynamic allocation currently does not support cached RDDs. Cached data for RDD " +
1402+
s"${rdd.id} will be lost when executors are removed.")
1403+
}
13991404
persistentRdds(rdd.id) = rdd
14001405
}
14011406

0 commit comments

Comments
 (0)