Skip to content

[SPARK-4000][BUILD] Sends archived unit tests logs to Jenkins master #2845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-4000][BUILD] Sends archived unit tests logs to Jenkins master #2845

wants to merge 3 commits into from

Conversation

liancheng
Copy link
Contributor

This PR sends archived unit tests logs to the build history directory in Jenkins master, so that we can serve it via HTTP later to help debugging Jenkins build failures.

@pwendell @JoshRosen Please help review, thanks!

@SparkQA
Copy link

SparkQA commented Oct 19, 2014

QA tests have started for PR 2845 at commit 4b912f7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 19, 2014

QA tests have finished for PR 2845 at commit 4b912f7.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21884/
Test FAILed.

liancheng
liancheng reviewed Oct 19, 2014
View reviewed changes
dev/run-tests-jenkins
echo "$log_files" | xargs tar czf ${log_archive}

local jenkins_build_dir=${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}
local scp_output=$(scp ${log_archive} amp-jenkins-master:${jenkins_build_dir}/${log_archive})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not good to hardcode Jenkins master hostname here. Should inject an extra environment variable $MASTER_NODE_NAME in Jenkins configurations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused actually - is "amp-jenkins-master" the current hostname of the master machine?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hostname is accessible from Jenkins slave nodes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liancheng I should be able to modify our pull request builders to inject this variable as JENKINS_MASTER_HOSTNAME.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks :)

@SparkQA
Copy link

SparkQA commented Oct 19, 2014

QA tests have started for PR 2845 at commit 68c7010.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 19, 2014

QA tests have finished for PR 2845 at commit 68c7010.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21889/
Test PASSed.

@liancheng
Copy link
Contributor Author

Verified that the log archive was uploaded to the correct location in Jenkins master node.

pwendell
pwendell reviewed Oct 19, 2014
View reviewed changes
dev/run-tests-jenkins
echo "> No log files found." >&2
else
local log_archive="unit-tests-logs.tar.gz"
echo "$log_files" | xargs tar czf ${log_archive}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, will these appear in the tarfile under the full path (e.g. streaming/target/unit-tests.log)? That's ideal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@pwendell
Copy link
Contributor

Overall this looks good, had a few minor questions. One thing we can do next is that we could scp the logs to a web server that we control (e.g. something under people.apache.org) and clean up the old ones every time we copy something over.

@SparkQA
Copy link

SparkQA commented Oct 20, 2014

Test build #21898 has started for PR 2845 at commit ac8d9d4.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 20, 2014

Test build #21898 has finished for PR 2845 at commit ac8d9d4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21898/
Test PASSed.

@liancheng
Copy link
Contributor Author

@pwendell This should be ready to go.

@JoshRosen JoshRosen mentioned this pull request Oct 24, 2014
Closed
@JoshRosen
Copy link
Contributor

This looks good to me. @pwendell, any more feedback? I'd like to get this merged soon in order to help debug a few Jenkins failures.

@JoshRosen
Copy link
Contributor

Alright, I've modified the pull request builder job to inject the master hostname as JENKINS_MASTER_HOSTNAME.

@pwendell
Copy link
Contributor

Merge away!

On Thu, Oct 23, 2014 at 10:10 PM, Josh Rosen [email protected]
wrote:

Alright, I've modified the pull request builder job to inject the master
hostname as JENKINS_MASTER_HOSTNAME.

—
Reply to this email directly or view it on GitHub
#2845 (comment).

@JoshRosen
Copy link
Contributor

Alright, I'm going to merge this now and I'll hotfix the JENKINS_MASTER_HOSTNAME myself later.

@asfgit asfgit closed this in a29c9bd Oct 24, 2014
@liancheng liancheng deleted the log-archive branch October 24, 2014 05:45
@JoshRosen
Copy link
Contributor

Shortcut / reminder for folks like me who don't know where JEKNINS_HOME is offhand: the paths to individual logs are like

/var/lib/jenkins/jobs/SparkPullRequestBuilder/builds/2014-10-24_09-52-18

@liancheng liancheng mentioned this pull request Nov 9, 2014
Closed
guavuslabs-builder pushed a commit to ThalesGroup/spark that referenced this pull request Nov 11, 2014
@liancheng @marmbrus
[SPARK-4000][Build] Uploads HiveCompatibilitySuite logs
534b231 
This is a follow up of apache#2845. In addition to unit-tests.log files, also upload failure output files generated by `HiveCompatibilitySuite` to Jenkins master. These files can be very helpful to debug Hive compatibility test failures.

/cc pwendell marmbrus

Author: Cheng Lian <[email protected]>

Closes apache#2993 from liancheng/upload-hive-compat-logs and squashes the following commits:

8e6247f [Cheng Lian] Uploads HiveCompatibilitySuite logs
tianyi pushed a commit to asiainfo/spark that referenced this pull request Dec 4, 2014
@liancheng @tianyi
[SPARK-4000][Build] Uploads HiveCompatibilitySuite logs
a53ac8f 
This is a follow up of apache#2845. In addition to unit-tests.log files, also upload failure output files generated by `HiveCompatibilitySuite` to Jenkins master. These files can be very helpful to debug Hive compatibility test failures.

/cc pwendell marmbrus

Author: Cheng Lian <[email protected]>

Closes apache#2993 from liancheng/upload-hive-compat-logs and squashes the following commits:

8e6247f [Cheng Lian] Uploads HiveCompatibilitySuite logs
tianyi pushed a commit to asiainfo/spark that referenced this pull request Dec 4, 2014
@liancheng @tianyi
[SPARK-4000][Build] Uploads HiveCompatibilitySuite logs
40a7b28 
This is a follow up of apache#2845. In addition to unit-tests.log files, also upload failure output files generated by `HiveCompatibilitySuite` to Jenkins master. These files can be very helpful to debug Hive compatibility test failures.

/cc pwendell marmbrus

Author: Cheng Lian <[email protected]>

Closes apache#2993 from liancheng/upload-hive-compat-logs and squashes the following commits:

8e6247f [Cheng Lian] Uploads HiveCompatibilitySuite logs
tianyi pushed a commit to asiainfo/spark that referenced this pull request Dec 4, 2014
@liancheng @tianyi
[SPARK-4000][Build] Uploads HiveCompatibilitySuite logs
c0e631a 
This is a follow up of apache#2845. In addition to unit-tests.log files, also upload failure output files generated by `HiveCompatibilitySuite` to Jenkins master. These files can be very helpful to debug Hive compatibility test failures.

/cc pwendell marmbrus

Author: Cheng Lian <[email protected]>

Closes apache#2993 from liancheng/upload-hive-compat-logs and squashes the following commits:

8e6247f [Cheng Lian] Uploads HiveCompatibilitySuite logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@liancheng @SparkQA @AmplabJenkins @pwendell @JoshRosen @nchammas