Skip to content

[SPARK-7657][YARN] Add driver logs links in application UI, in cluster mode. #6166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 14 commits into from
Closed

[SPARK-7657][YARN] Add driver logs links in application UI, in cluster mode. #6166

wants to merge 14 commits into from

Conversation

harishreedharan
Copy link
Contributor

This PR adds the URLs to the driver logs to SparkListenerApplicationStarted event, which is later used by the ExecutorsListener to populate the URLs to the driver logs in its own state. This info is then used when the UI is rendered to display links to the logs.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32762 has started for PR 6166 at commit 1cf338f.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32762 has finished for PR 6166 at commit 1cf338f.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32762/
Test FAILed.

@harishreedharan
Copy link
Contributor Author

Looks like this API was added in 2.4.0. I don't see a way of adding this in older versions of YARN

@harishreedharan
Copy link
Contributor Author

Looks like this can be fixed using NodeReport which was available in Hadoop 2.2. Testing it right now.

@harishreedharan
Use NodeReport to get the URL for the logs. Also, make the environm…
402e8e4 
…ent variables generic so other cluster managers can use them as well.
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@harishreedharan harishreedharan changed the title [SPARK-7657][YARN] Add driver link in application UI, in cluster mode. [SPARK-7657][YARN] Add driver logs links in application UI, in cluster mode. May 15, 2015
@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32784 has started for PR 6166 at commit 402e8e4.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32784 has finished for PR 6166 at commit 402e8e4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32784/
Test PASSed.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32807 has started for PR 6166 at commit 50cdae3.

@harishreedharan
Copy link
Contributor Author

@vanzin @squito @tgraves

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32807 has finished for PR 6166 at commit 50cdae3.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32807/
Test FAILed.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32835 has started for PR 6166 at commit c0de336.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32835 has finished for PR 6166 at commit c0de336.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33293/
Test PASSed.

@harishreedharan
Copy link
Contributor Author

Jenkins, test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33303 has started for PR 6166 at commit 943fc4f.

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33295 has finished for PR 6166 at commit 943fc4f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class SparkListenerApplicationStart(
    • s" but class priors vector pi had $
    • s" but class conditionals array theta had $
    • class KernelDensity extends Serializable

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33295/
Test PASSed.

@harishreedharan
Copy link
Contributor Author

Jenkins, test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33307 has started for PR 6166 at commit 943fc4f.

@asfgit asfgit closed this in 956c4c9 May 22, 2015
@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33303 has finished for PR 6166 at commit 943fc4f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33303/
Test PASSed.

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33307 has finished for PR 6166 at commit 943fc4f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class SparkListenerApplicationStart(

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33307/
Test PASSed.

@andrewor14
Copy link
Contributor

@harishreedharan I wasn't able to review this in time but for breaking changes like these you need to add a backward compatibility test in JsonProtocolSuite. Could you submit a hot fix to add it?

@harishreedharan
Copy link
Contributor Author

@andrewor14 Sure, will do.

@harishreedharan
Copy link
Contributor Author

@andrewor14 The current test for SparkListenerApplicationStart already acts as a regression test for events without driver logs. I will add a test for events with driver logs.

@harishreedharan harishreedharan mentioned this pull request May 22, 2015
Closed
asfgit pushed a commit that referenced this pull request May 23, 2015
@harishreedharan
[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
368b8c2 
#6166 added the driver logs to `SparkListenerApplicationStart`. This  adds tests in `JsonProtocolSuite` to ensure we don't regress.

Author: Hari Shreedharan <[email protected]>

Closes #6368 from harishreedharan/jsonprotocol-test and squashes the following commits:

dc9eafc [Hari Shreedharan] [HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@harishreedharan @jeanlyn
[SPARK-7657] [YARN] Add driver logs links in application UI, in clust…
2d2b455 
…er mode.

This PR adds the URLs to the driver logs to `SparkListenerApplicationStarted` event, which is later used by the `ExecutorsListener` to populate the URLs to the driver logs in its own state. This info is then used when the UI is rendered to display links to the logs.

Author: Hari Shreedharan <[email protected]>

Closes apache#6166 from harishreedharan/am-log-link and squashes the following commits:

943fc4f [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
9e5c04b [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
b3f9b9d [Hari Shreedharan] Updated comment based on feedback.
0840a95 [Hari Shreedharan] Move the result and sc.stop back to original location, minor import changes.
537a2f7 [Hari Shreedharan] Add test to ensure the log urls are populated and valid.
4033725 [Hari Shreedharan] Adding comments explaining how node reports are used to get the log urls.
6c5c285 [Hari Shreedharan] Import order.
346f4ea [Hari Shreedharan] Review feedback fixes.
629c1dc [Hari Shreedharan] Cleanup.
99fb1a3 [Hari Shreedharan] Send the log urls in App start event, to ensure that other listeners are not affected.
c0de336 [Hari Shreedharan] Ensure new unit test cleans up after itself.
50cdae3 [Hari Shreedharan] Added unit test, made the approach generic.
402e8e4 [Hari Shreedharan] Use `NodeReport` to get the URL for the logs. Also, make the environment variables generic so other cluster managers can use them as well.
1cf338f [Hari Shreedharan] [SPARK-7657][YARN] Add driver link in application UI, in cluster mode.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@harishreedharan @jeanlyn
[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
deb61cc 
apache#6166 added the driver logs to `SparkListenerApplicationStart`. This  adds tests in `JsonProtocolSuite` to ensure we don't regress.

Author: Hari Shreedharan <[email protected]>

Closes apache#6368 from harishreedharan/jsonprotocol-test and squashes the following commits:

dc9eafc [Hari Shreedharan] [HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@harishreedharan @jeanlyn
[SPARK-7657] [YARN] Add driver logs links in application UI, in clust…
04d379e 
…er mode.

This PR adds the URLs to the driver logs to `SparkListenerApplicationStarted` event, which is later used by the `ExecutorsListener` to populate the URLs to the driver logs in its own state. This info is then used when the UI is rendered to display links to the logs.

Author: Hari Shreedharan <[email protected]>

Closes apache#6166 from harishreedharan/am-log-link and squashes the following commits:

943fc4f [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
9e5c04b [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
b3f9b9d [Hari Shreedharan] Updated comment based on feedback.
0840a95 [Hari Shreedharan] Move the result and sc.stop back to original location, minor import changes.
537a2f7 [Hari Shreedharan] Add test to ensure the log urls are populated and valid.
4033725 [Hari Shreedharan] Adding comments explaining how node reports are used to get the log urls.
6c5c285 [Hari Shreedharan] Import order.
346f4ea [Hari Shreedharan] Review feedback fixes.
629c1dc [Hari Shreedharan] Cleanup.
99fb1a3 [Hari Shreedharan] Send the log urls in App start event, to ensure that other listeners are not affected.
c0de336 [Hari Shreedharan] Ensure new unit test cleans up after itself.
50cdae3 [Hari Shreedharan] Added unit test, made the approach generic.
402e8e4 [Hari Shreedharan] Use `NodeReport` to get the URL for the logs. Also, make the environment variables generic so other cluster managers can use them as well.
1cf338f [Hari Shreedharan] [SPARK-7657][YARN] Add driver link in application UI, in cluster mode.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@harishreedharan @jeanlyn
[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
8b3b1de 
apache#6166 added the driver logs to `SparkListenerApplicationStart`. This  adds tests in `JsonProtocolSuite` to ensure we don't regress.

Author: Hari Shreedharan <[email protected]>

Closes apache#6368 from harishreedharan/jsonprotocol-test and squashes the following commits:

dc9eafc [Hari Shreedharan] [HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@harishreedharan @nemccarthy
[SPARK-7657] [YARN] Add driver logs links in application UI, in clust…
c22458a 
…er mode.

This PR adds the URLs to the driver logs to `SparkListenerApplicationStarted` event, which is later used by the `ExecutorsListener` to populate the URLs to the driver logs in its own state. This info is then used when the UI is rendered to display links to the logs.

Author: Hari Shreedharan <[email protected]>

Closes apache#6166 from harishreedharan/am-log-link and squashes the following commits:

943fc4f [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
9e5c04b [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into am-log-link
b3f9b9d [Hari Shreedharan] Updated comment based on feedback.
0840a95 [Hari Shreedharan] Move the result and sc.stop back to original location, minor import changes.
537a2f7 [Hari Shreedharan] Add test to ensure the log urls are populated and valid.
4033725 [Hari Shreedharan] Adding comments explaining how node reports are used to get the log urls.
6c5c285 [Hari Shreedharan] Import order.
346f4ea [Hari Shreedharan] Review feedback fixes.
629c1dc [Hari Shreedharan] Cleanup.
99fb1a3 [Hari Shreedharan] Send the log urls in App start event, to ensure that other listeners are not affected.
c0de336 [Hari Shreedharan] Ensure new unit test cleans up after itself.
50cdae3 [Hari Shreedharan] Added unit test, made the approach generic.
402e8e4 [Hari Shreedharan] Use `NodeReport` to get the URL for the logs. Also, make the environment variables generic so other cluster managers can use them as well.
1cf338f [Hari Shreedharan] [SPARK-7657][YARN] Add driver link in application UI, in cluster mode.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@harishreedharan @nemccarthy
[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
8d3c460 
apache#6166 added the driver logs to `SparkListenerApplicationStart`. This  adds tests in `JsonProtocolSuite` to ensure we don't regress.

Author: Hari Shreedharan <[email protected]>

Closes apache#6368 from harishreedharan/jsonprotocol-test and squashes the following commits:

dc9eafc [Hari Shreedharan] [HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@harishreedharan @AmplabJenkins @SparkQA @vanzin @squito @andrewor14