[SPARK-7224] added mock repository generator for --packages tests #5790

brkyvz · 2015-04-29T21:54:20Z

This patch contains an IvyTestUtils file, which dynamically generates jars and pom files to test the --packages feature without having to rely on the internet, and Maven Central.

cc @pwendell I know that there existed Util functions to create Jars and stuff already, but they didn't really serve my purposes as they appended random prefixes that was breaking things.

I also added the local repository tests. Notice that they work without passing the repo to resolveMavenCoordinates.

pwendell · 2015-04-29T22:02:51Z

core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala

+  }
+
+  /** Create a simple testable Class. */
+  private def createJavaClass(dir: File, className: String, packageName: String): File = {


I think this is duplicating some of the logic in:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/TestUtils.scala#L114

oh yeah, forgot to remove that. Thanks!

SparkQA · 2015-04-30T01:06:19Z

Test build #31348 has finished for PR 5790 at commit a39151b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

pwendell · 2015-04-30T07:56:53Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -731,6 +734,10 @@ private[deploy] object SparkSubmitUtils {
    }
  }

+  /** Path of the local Maven cache. */
+  def m2Path: JavaPath = new File(System.getProperty("user.home"),


make this private spark

SparkQA · 2015-04-30T16:32:10Z

Test build #31425 has finished for PR 5790 at commit 3ec79b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable
This patch does not change any dependencies.

pwendell · 2015-04-30T17:16:40Z

Looks good - thanks I can merge this.

tgravescs · 2015-05-01T14:15:02Z

@pwendell @brkyvz unfortunately java.nio.file.Path (used in core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala) is a jdk7 only api. It breaks the build when using jdk6. Can we fix this to not use or revert?

tgravescs · 2015-05-01T18:22:33Z

filed https://issues.apache.org/jira/browse/SPARK-7306

pwendell · 2015-05-01T19:59:46Z

Thanks tom I reverted this.

On Fri, May 1, 2015 at 11:23 AM, Tom Graves [email protected]
wrote:

filed https://issues.apache.org/jira/browse/SPARK-7306

—
Reply to this email directly or view it on GitHub
#5790 (comment).

…sts without nio.Path The previous PR for SPARK-7224 (#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes #5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator (cherry picked from commit 8014e1f) Signed-off-by: Josh Rosen <[email protected]>

…sts without nio.Path The previous PR for SPARK-7224 (#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes #5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I know that there existed Util functions to create Jars and stuff already, but they didn't really serve my purposes as they appended random prefixes that was breaking things. I also added the local repository tests. Notice that they work without passing the `repo` to `resolveMavenCoordinates`. Author: Burak Yavuz <[email protected]> Closes apache#5790 from brkyvz/maven-utils and squashes the following commits: 3ec79b7 [Burak Yavuz] addressed comments v0.2 a39151b [Burak Yavuz] address comments v0.1 172dfef [Burak Yavuz] use Ivy format 7476d06 [Burak Yavuz] added mock repository generator

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala core/src/test/scala/org/apache/spark/deploy/SparkSubmitUtilsSuite.scala

This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I know that there existed Util functions to create Jars and stuff already, but they didn't really serve my purposes as they appended random prefixes that was breaking things. I also added the local repository tests. Notice that they work without passing the `repo` to `resolveMavenCoordinates`. Author: Burak Yavuz <[email protected]> Closes apache#5790 from brkyvz/maven-utils and squashes the following commits: 3ec79b7 [Burak Yavuz] addressed comments v0.2 a39151b [Burak Yavuz] address comments v0.1 172dfef [Burak Yavuz] use Ivy format 7476d06 [Burak Yavuz] added mock repository generator

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I know that there existed Util functions to create Jars and stuff already, but they didn't really serve my purposes as they appended random prefixes that was breaking things. I also added the local repository tests. Notice that they work without passing the `repo` to `resolveMavenCoordinates`. Author: Burak Yavuz <[email protected]> Closes apache#5790 from brkyvz/maven-utils and squashes the following commits: 3ec79b7 [Burak Yavuz] addressed comments v0.2 a39151b [Burak Yavuz] address comments v0.1 172dfef [Burak Yavuz] use Ivy format 7476d06 [Burak Yavuz] added mock repository generator

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <[email protected]> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

brkyvz added 2 commits April 29, 2015 14:46

added mock repository generator

7476d06

use Ivy format

172dfef

pwendell reviewed Apr 29, 2015
View reviewed changes

address comments v0.1

a39151b

pwendell reviewed Apr 30, 2015
View reviewed changes

addressed comments v0.2

3ec79b7

asfgit closed this in 7dacc08 Apr 30, 2015

brkyvz mentioned this pull request May 4, 2015

[SPARK-7224][SPARK-7306] mock repository generator for --packages tests without nio.Path #5892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-7224] added mock repository generator for --packages tests #5790

[SPARK-7224] added mock repository generator for --packages tests #5790

Uh oh!

brkyvz commented Apr 29, 2015

Uh oh!

pwendell Apr 29, 2015

Uh oh!

brkyvz Apr 29, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

pwendell Apr 30, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

pwendell commented Apr 30, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

pwendell commented May 1, 2015

Uh oh!

Uh oh!

[SPARK-7224] added mock repository generator for --packages tests #5790

[SPARK-7224] added mock repository generator for --packages tests #5790

Uh oh!

Conversation

brkyvz commented Apr 29, 2015

Uh oh!

pwendell Apr 29, 2015

Choose a reason for hiding this comment

Uh oh!

brkyvz Apr 29, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

pwendell Apr 30, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

pwendell commented Apr 30, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

pwendell commented May 1, 2015

Uh oh!

Uh oh!