Skip to content

Commit 4ecf841

Browse files
committed
Make proactive serializability checking optional.
SparkContext.clean uses ClosureCleaner's proactive serializability checking by default. This commit adds an overloaded clean method to SparkContext that allows clients to specify that serializability checking should not occur as part of closure cleaning.
1 parent d8df3db commit 4ecf841

File tree

1 file changed

+9
-1
lines changed

1 file changed

+9
-1
lines changed

core/src/main/scala/org/apache/spark/SparkContext.scala

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1032,7 +1032,15 @@ class SparkContext(
10321032
* (removes unreferenced variables in $outer's, updates REPL variables)
10331033
*/
10341034
private[spark] def clean[F <: AnyRef](f: F): F = {
1035-
ClosureCleaner.clean(f)
1035+
clean(f, true)
1036+
}
1037+
1038+
/**
1039+
* Clean a closure to make it ready to serialized and send to tasks
1040+
* (removes unreferenced variables in $outer's, updates REPL variables)
1041+
*/
1042+
private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean): F = {
1043+
ClosureCleaner.clean(f, checkSerializable)
10361044
f
10371045
}
10381046

0 commit comments

Comments
 (0)