[SPARK-4172] [PySpark] Progress API in Python #3027

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

AmplabJenkins · 2014-11-19T22:27:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23628/
Test FAILed.

SparkQA · 2014-11-24T21:30:32Z

Test build #534 has started for PR 3027 at commit 25590c9.

This patch merges cleanly.

SparkQA · 2014-11-24T22:57:09Z

Test build #534 has finished for PR 3027 at commit 25590c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

Conflicts: python/pyspark/__init__.py python/pyspark/context.py

SparkQA · 2015-01-29T22:57:48Z

Test build #26342 has started for PR 3027 at commit 969fa9d.

This patch merges cleanly.

SparkQA · 2015-01-30T00:06:12Z

Test build #26342 has finished for PR 3027 at commit 969fa9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

AmplabJenkins · 2015-01-30T00:06:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26342/
Test PASSed.

SparkQA · 2015-02-17T06:27:30Z

Test build #27614 has started for PR 3027 at commit d3b9253.

This patch merges cleanly.

SparkQA · 2015-02-17T06:28:55Z

Test build #27614 has finished for PR 3027 at commit d3b9253.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

AmplabJenkins · 2015-02-17T06:28:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27614/
Test FAILed.

JoshRosen · 2015-02-17T19:07:00Z

Looks like this failed due to Scala style issues due to a few long lines.

davies · 2015-02-17T19:09:58Z

@JoshRosen I fixed it locally, but forgot to push out.

SparkQA · 2015-02-17T19:10:32Z

Test build #612 has started for PR 3027 at commit d3b9253.

This patch merges cleanly.

davies · 2015-02-17T19:11:10Z

core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala

-          if (serializedData != null && serializedData.limit() > 0) {
-            reason = serializer.get().deserialize[TaskEndReason](
-              serializedData, Utils.getSparkClassLoader)
+    try {


This is not related change, it mute the exception when you cancel a job.

SparkQA · 2015-02-17T19:12:29Z

Test build #27637 has started for PR 3027 at commit b1ba984.

This patch merges cleanly.

SparkQA · 2015-02-17T20:22:30Z

Test build #612 has finished for PR 3027 at commit d3b9253.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

SparkQA · 2015-02-17T20:26:15Z

Test build #27637 has finished for PR 3027 at commit b1ba984.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

AmplabJenkins · 2015-02-17T20:26:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27637/
Test PASSed.

JoshRosen · 2015-02-17T21:36:25Z

LGTM, so I'm going to merge this into master (1.4.0) and branch-1.3 (1.3.0). Thanks!

This patch bring the pull based progress API into Python, also a example in Python. Author: Davies Liu <[email protected]> Closes #3027 from davies/progress_api and squashes the following commits: b1ba984 [Davies Liu] fix style d3b9253 [Davies Liu] add tests, mute the exception after stop 4297327 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress_api 969fa9d [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress_api 25590c9 [Davies Liu] update with Java API 360de2d [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress_api c0f1021 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress_api 023afb3 [Davies Liu] add Python API and example for progress API (cherry picked from commit 445a755) Signed-off-by: Josh Rosen <[email protected]>

Davies Liu added 2 commits October 27, 2014 15:15

add Python API and example for progress API

023afb3

Merge branch 'master' of github.com:apache/spark into progress_api

c0f1021

davies changed the title ~~[SPARK-2312] [PySpark] Progress API in Python~~ [SPARK-4172] [PySpark] Progress API in Python Oct 31, 2014

davies changed the title ~~[SPARK-4172] [PySpark] Progress API in Python~~ [WIP][SPARK-4172] [PySpark] Progress API in Python Nov 19, 2014

Davies Liu added 2 commits November 19, 2014 12:08

Merge branch 'master' of github.com:apache/spark into progress_api

360de2d

update with Java API

25590c9

davies force-pushed the progress_api branch from c792134 to 25590c9 Compare November 19, 2014 20:58

davies changed the title ~~[WIP][SPARK-4172] [PySpark] Progress API in Python~~ [SPARK-4172] [PySpark] Progress API in Python Nov 19, 2014

Merge branch 'master' of github.com:apache/spark into progress_api

969fa9d

Conflicts: python/pyspark/__init__.py python/pyspark/context.py

Davies Liu added 2 commits February 16, 2015 14:40

Merge branch 'master' of github.com:apache/spark into progress_api

4297327

add tests, mute the exception after stop

d3b9253

fix style

b1ba984

davies reviewed Feb 17, 2015
View reviewed changes

davies mentioned this pull request Feb 17, 2015

[SPARK-5785] [PySpark] narrow dependency for cogroup/join in PySpark #4629

Closed

asfgit closed this in 445a755 Feb 17, 2015

[SPARK-4172] [PySpark] Progress API in Python #3027

[SPARK-4172] [PySpark] Progress API in Python #3027

Uh oh!

Conversation

davies commented Oct 30, 2014

Uh oh!

SparkQA commented Oct 30, 2014

Uh oh!

SparkQA commented Oct 31, 2014

Uh oh!

AmplabJenkins commented Oct 31, 2014

Uh oh!

JoshRosen commented Oct 31, 2014

Uh oh!

davies commented Nov 7, 2014

Uh oh!

SparkQA commented Nov 16, 2014

Uh oh!

SparkQA commented Nov 16, 2014

Uh oh!

davies commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 19, 2014

Uh oh!

AmplabJenkins commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

SparkQA commented Jan 29, 2015

Uh oh!

SparkQA commented Jan 30, 2015

Uh oh!

AmplabJenkins commented Jan 30, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

AmplabJenkins commented Feb 17, 2015

Uh oh!

JoshRosen commented Feb 17, 2015

Uh oh!

davies commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

davies Feb 17, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

AmplabJenkins commented Feb 17, 2015

Uh oh!

JoshRosen commented Feb 17, 2015

Uh oh!

Uh oh!