[SPARK-7237] [SPARK-7741] [Core] [Streaming] Clean more closures that need cleaning #6269

andrewor14 · 2015-05-19T21:40:38Z

SPARK-7741 is the equivalent of SPARK-7237 in streaming. This is an alternative to #6268.

AmplabJenkins · 2015-05-19T21:42:10Z

Merged build triggered.

AmplabJenkins · 2015-05-19T21:42:19Z

Merged build started.

SparkQA · 2015-05-19T21:43:03Z

Test build #33099 has started for PR 6269 at commit 5431f61.

SparkQA · 2015-05-19T23:31:03Z

Test build #33099 has finished for PR 6269 at commit 5431f61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-19T23:31:08Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-19T23:31:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33099/
Test PASSed.

tdas · 2015-05-19T23:48:09Z

foreachRDD is missing.

andrewor14 · 2015-05-19T23:56:49Z

Good catch

AmplabJenkins · 2015-05-20T00:02:11Z

Merged build triggered.

AmplabJenkins · 2015-05-20T00:02:20Z

Merged build started.

SparkQA · 2015-05-20T00:04:11Z

Test build #33106 has started for PR 6269 at commit 328139b.

SparkQA · 2015-05-20T01:19:19Z

Test build #33106 has finished for PR 6269 at commit 328139b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-20T01:19:24Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T01:19:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33106/
Test FAILed.

andrewor14 · 2015-05-20T01:51:04Z

Okay, I added tests to verify that closures are being cleaned in DStream operations. @tdas PTAL.

AmplabJenkins · 2015-05-20T01:52:10Z

Merged build triggered.

AmplabJenkins · 2015-05-20T01:52:16Z

Merged build started.

SparkQA · 2015-05-20T01:54:22Z

Test build #33117 has started for PR 6269 at commit d18c9f9.

andrewor14 · 2015-05-20T01:55:13Z

Regarding foreachRDD, I'm actually pretty confused about one thing. Currently when we clean the foreach closure we don't check serializability for the following reason:

// because the DStream is reachable from the outer object here, and because 
// DStreams can't be serialized with closures, we can't proactively check 
// it for serializability and so we pass the optional false to SparkContext.clean
new ForEachDStream(this, context.sparkContext.clean(foreachFunc, false)).register()

This implies that the closure cannot be serialized ever, which is true because we pull in the whole ssc into the closure in ForEachInputDStream#generateJob:

val jobFunc = () => createRDDWithLocalProperties(time) {
  ssc.sparkContext.setCallSite(creationSite)
  foreachFunc(rdd, time)
}

So, my question is the following:
What's the point in cleaning this closure if it can never be made serializable?

(By the way, this is a separate issue that we can address later. For now I'm OK with just not checking serializability in foreachRDD, which is arbitrary but doesn't really hurt.)

SparkQA · 2015-05-20T02:05:06Z

Test build #33117 has finished for PR 6269 at commit d18c9f9.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SimpleFunctionRegistry(val conf: CatalystConf) extends FunctionRegistry

AmplabJenkins · 2015-05-20T02:05:10Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T02:05:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33117/
Test FAILed.

andrewor14 · 2015-05-20T03:34:29Z

retest this please

AmplabJenkins · 2015-05-20T06:07:14Z

Merged build triggered.

AmplabJenkins · 2015-05-20T06:07:24Z

Merged build started.

SparkQA · 2015-05-20T06:09:21Z

Test build #33127 has started for PR 6269 at commit 79a435b.

SparkQA · 2015-05-20T07:57:36Z

Test build #33127 has finished for PR 6269 at commit 79a435b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-20T07:57:40Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-20T07:57:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33127/
Test PASSed.

AmplabJenkins · 2015-05-20T18:52:15Z

Merged build triggered.

AmplabJenkins · 2015-05-20T18:52:23Z

Merged build started.

SparkQA · 2015-05-20T18:53:12Z

Test build #33160 has started for PR 6269 at commit c51c9ab.

SparkQA · 2015-05-20T20:05:35Z

Test build #33160 has finished for PR 6269 at commit c51c9ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilabelMetrics(JavaModelWrapper):
- class GroupedData protected[sql](

AmplabJenkins · 2015-05-20T20:05:41Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T20:05:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33160/
Test FAILed.

andrewor14 · 2015-05-20T20:27:48Z

retest this please

tdas · 2015-05-20T20:29:05Z

LGTM!

AmplabJenkins · 2015-05-20T20:32:13Z

Merged build triggered.

AmplabJenkins · 2015-05-20T20:32:19Z

Merged build started.

SparkQA · 2015-05-20T20:32:58Z

Test build #33169 has started for PR 6269 at commit c51c9ab.

SparkQA · 2015-05-20T22:22:07Z

Test build #33169 has finished for PR 6269 at commit c51c9ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-20T22:22:11Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-20T22:22:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33169/
Test PASSed.

… need cleaning SPARK-7741 is the equivalent of SPARK-7237 in streaming. This is an alternative to #6268. Author: Andrew Or <[email protected]> Closes #6269 from andrewor14/clean-moar and squashes the following commits: c51c9ab [Andrew Or] Add periods (trivial) 6c686ac [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-moar 79a435b [Andrew Or] Fix tests d18c9f9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-moar 65ef07b [Andrew Or] Fix tests? 4b487a3 [Andrew Or] Add tests for closures passed to DStream operations 328139b [Andrew Or] Do not forget foreachRDD 5431f61 [Andrew Or] Clean streaming closures 72b7b73 [Andrew Or] Clean core closures (cherry picked from commit 9b84443) Signed-off-by: Tathagata Das <[email protected]>

… need cleaning SPARK-7741 is the equivalent of SPARK-7237 in streaming. This is an alternative to apache#6268. Author: Andrew Or <[email protected]> Closes apache#6269 from andrewor14/clean-moar and squashes the following commits: c51c9ab [Andrew Or] Add periods (trivial) 6c686ac [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-moar 79a435b [Andrew Or] Fix tests d18c9f9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-moar 65ef07b [Andrew Or] Fix tests? 4b487a3 [Andrew Or] Add tests for closures passed to DStream operations 328139b [Andrew Or] Do not forget foreachRDD 5431f61 [Andrew Or] Clean streaming closures 72b7b73 [Andrew Or] Clean core closures

Andrew Or added 2 commits May 19, 2015 14:32

Clean core closures

72b7b73

Clean streaming closures

5431f61

andrewor14 mentioned this pull request May 19, 2015

[SPARK-7741] [STREAMING] Added sparkContext.clean at missing places #6268

Closed

Do not forget foreachRDD

328139b

Andrew Or added 3 commits May 19, 2015 18:36

Add tests for closures passed to DStream operations

4b487a3

Fix tests?

65ef07b

Merge branch 'master' of github.com:apache/spark into clean-moar

d18c9f9

Fix tests

79a435b

Andrew Or added 2 commits May 20, 2015 11:47

Merge branch 'master' of github.com:apache/spark into clean-moar

6c686ac

Add periods (trivial)

c51c9ab

andrewor14 changed the title ~~[SPARK-7237] [SPARK-7741] Clean more closures that need cleaning~~ [SPARK-7237] [SPARK-7741] [Core] [Streaming] Clean more closures that need cleaning May 20, 2015

asfgit closed this in 9b84443 May 20, 2015

andrewor14 deleted the clean-moar branch May 20, 2015 23:27

[SPARK-7237] [SPARK-7741] [Core] [Streaming] Clean more closures that need cleaning #6269

[SPARK-7237] [SPARK-7741] [Core] [Streaming] Clean more closures that need cleaning #6269

Uh oh!

Conversation

andrewor14 commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

tdas commented May 19, 2015

Uh oh!

andrewor14 commented May 19, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

andrewor14 commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

andrewor14 commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

andrewor14 commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

andrewor14 commented May 20, 2015

Uh oh!

tdas commented May 20, 2015

Uh oh!