Skip to content

Commit

Permalink
Make proactive serializability checking optional.
Browse files Browse the repository at this point in the history
SparkContext.clean uses ClosureCleaner's proactive serializability
checking by default.  This commit adds an overloaded clean method
to SparkContext that allows clients to specify that serializability
checking should not occur as part of closure cleaning.
  • Loading branch information
willb committed Mar 20, 2014
1 parent d8df3db commit 4ecf841
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion core/src/main/scala/org/apache/spark/SparkContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1032,7 +1032,15 @@ class SparkContext(
* (removes unreferenced variables in $outer's, updates REPL variables)
*/
private[spark] def clean[F <: AnyRef](f: F): F = {
ClosureCleaner.clean(f)
clean(f, true)
}

/**
* Clean a closure to make it ready to serialized and send to tasks
* (removes unreferenced variables in $outer's, updates REPL variables)
*/
private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean): F = {
ClosureCleaner.clean(f, checkSerializable)
f
}

Expand Down

0 comments on commit 4ecf841

Please sign in to comment.