[SPARK-1870] Made deployment with --jars work in yarn-standalone mode. #1013

dbtsai · 2014-06-08T23:22:23Z

Ported from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start.

AmplabJenkins · 2014-06-08T23:22:49Z

Merged build triggered.

AmplabJenkins · 2014-06-08T23:22:57Z

Merged build started.

dbtsai · 2014-06-08T23:23:37Z

CC: @mengxr and @sryza

AmplabJenkins · 2014-06-08T23:51:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-08T23:51:46Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15549/

mengxr · 2014-06-09T04:04:00Z

yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

    // Normally the users app.jar is last in case conflicts with spark jars
    val userClasspathFirst = sparkConf.get("spark.yarn.user.classpath.first", "false").toBoolean
    if (userClasspathFirst) {
      Apps.addToEnvironment(env, Environment.CLASSPATH.name, Environment.PWD.$() +
        Path.SEPARATOR + APP_JAR)
+      cachedSecondaryJarLinks.foreach(jarLink =>
+        Apps.addToEnvironment(env, Environment.CLASSPATH.name, Environment.PWD.$() +
+        Path.SEPARATOR + jarLink))


minor: add two more spaces.

mengxr · 2014-06-09T04:05:15Z

@dbtsai Thanks for fixing it in branch-0.9! Did you verify the patch on a YARN cluster?

dbtsai · 2014-06-09T04:14:22Z

Work in my local VM. Should work in real yarn cluster. Will test it tomorrow in the office.

pwendell · 2014-06-09T18:44:15Z

@dbtsai could you update the title of this PR to make it more clear what issue this relates to?

AmplabJenkins · 2014-06-09T21:02:50Z

Merged build triggered.

AmplabJenkins · 2014-06-09T22:05:43Z

Merged build started.

dbtsai · 2014-06-09T22:16:48Z

Tested in PivotalHD 1.1 Yarn 4 node cluster. With --addjars file:///somePath/to/jar, launching spark application works.

AmplabJenkins · 2014-06-09T22:37:49Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-09T22:37:49Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15578/

andrewor14 · 2014-06-10T00:59:23Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -281,18 +280,19 @@ class Client(args: ClientArguments, conf: Configuration, sparkConf: SparkConf)
    }

    // Handle jars local to the ApplicationMaster.
+    var cachedSecondaryJarLinks = ListBuffer.empty[String]


I think this can be a val

andrewor14 · 2014-06-10T01:02:22Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -507,12 +508,19 @@ object Client {
      Apps.addToEnvironment(env, Environment.CLASSPATH.name, Environment.PWD.$() +
        Path.SEPARATOR + LOG4J_PROP)
    }
+
+    val cachedSecondaryJarLinks =
+      sparkConf.getOption(CONF_SPARK_YARN_SECONDARY_JARS).getOrElse("").split(",")


This actually returns a one element Array("") if there are no secondary jars. We should .filter(_.nonEmpty) after splitting.

Thanks. You are right. It will add empty string to array, and then add the folder without file into classpath. Will fix in master as well.

AmplabJenkins · 2014-06-10T01:02:49Z

Merged build triggered.

andrewor14 · 2014-06-10T01:04:35Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

+
+    val cachedSecondaryJarLinks =
+      sparkConf.getOption(CONF_SPARK_YARN_SECONDARY_JARS).getOrElse("").split(",")
+
    // Normally the users app.jar is last in case conflicts with spark jars
    val userClasspathFirst = sparkConf.get("spark.yarn.user.classpath.first", "false")
      .toBoolean


minor: Not your code, but could you bump .toBoolean up one line? I'm pretty sure it's under 100 characters.

AmplabJenkins · 2014-06-10T01:22:50Z

Merged build triggered.

AmplabJenkins · 2014-06-10T04:05:32Z

Merged build started.

AmplabJenkins · 2014-06-10T04:06:31Z

Merged build started.

AmplabJenkins · 2014-06-10T04:34:17Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-10T04:34:17Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15590/

AmplabJenkins · 2014-06-10T04:35:22Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-10T04:35:22Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15591/

mengxr · 2014-06-10T04:36:38Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -507,12 +508,18 @@ object Client {
      Apps.addToEnvironment(env, Environment.CLASSPATH.name, Environment.PWD.$() +
        Path.SEPARATOR + LOG4J_PROP)
    }
+
+    val cachedSecondaryJarLinks =
+      sparkConf.getOption(CONF_SPARK_YARN_SECONDARY_JARS).getOrElse("").split(",").filter(_.nonEmpty)


line too wide?

AmplabJenkins · 2014-06-10T04:47:49Z

Merged build triggered.

AmplabJenkins · 2014-06-10T05:08:15Z

Merged build started.

AmplabJenkins · 2014-06-10T05:08:18Z

Merged build finished.

AmplabJenkins · 2014-06-10T05:08:18Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15600/

mengxr · 2014-06-10T05:21:42Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-10T05:22:49Z

Merged build triggered.

AmplabJenkins · 2014-06-10T05:22:56Z

Merged build started.

AmplabJenkins · 2014-06-10T05:52:04Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-10T05:52:05Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15603/

mengxr · 2014-06-10T05:57:52Z

LGTM. Thanks!

Ported from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start. Author: DB Tsai <[email protected]> Closes #1013 from dbtsai/branch-0.9 and squashes the following commits: c5696f4 [DB Tsai] fix line too long b085f10 [DB Tsai] Make sure that empty string is filtered out when we get secondary jars 3cc1085 [DB Tsai] changed from var to val ab94aa1 [DB Tsai] Code formatting. 0956af9 [DB Tsai] Ported SPARK-1870 from 1.0 branch to 0.9 branch

Ported SPARK-1870 from 1.0 branch to 0.9 branch

0956af9

mengxr reviewed Jun 9, 2014
View reviewed changes

Code formatting.

ab94aa1

dbtsai changed the title ~~[SPARK-1870] Ported from 1.0 branch to 0.9 branch.~~ [SPARK-1870] Made deployment with --jars work in yarn-standalone mode. Jun 9, 2014

andrewor14 reviewed Jun 10, 2014
View reviewed changes

changed from var to val

3cc1085

andrewor14 reviewed Jun 10, 2014
View reviewed changes

Make sure that empty string is filtered out when we get secondary jars

b085f10

mengxr reviewed Jun 10, 2014
View reviewed changes

fix line too long

c5696f4

dbtsai closed this Jun 11, 2014

dbtsai deleted the branch-0.9 branch June 11, 2014 21:45

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-6061] Skip the storage detection for access-denied paths (#1013)

a16abaf

[SPARK-1870] Made deployment with --jars work in yarn-standalone mode. #1013

[SPARK-1870] Made deployment with --jars work in yarn-standalone mode. #1013

Uh oh!

Conversation

dbtsai commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

dbtsai commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

mengxr Jun 9, 2014

Choose a reason for hiding this comment

Uh oh!

mengxr commented Jun 9, 2014

Uh oh!

dbtsai commented Jun 9, 2014

Uh oh!

pwendell commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

dbtsai commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

andrewor14 Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

dbtsai Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

dbtsai Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

andrewor14 Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

mengxr Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

mengxr commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014