Skip to content

Commit 6f64a9b

Browse files
committed
Revert changes in YARN
There is currently no good way to handle quoted arguments and backslashes in YARN. The new code does not do any escaping, which is fine for standalone mode (which uses Java's ProcessBuilder) but not for YARN mode. I will open a separate JIRA for this.
1 parent 2f2908b commit 6f64a9b

File tree

2 files changed

+11
-2
lines changed

2 files changed

+11
-2
lines changed

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -386,7 +386,10 @@ trait ClientBase extends Logging {
386386
// Forward the Spark configuration to the application master / executors.
387387
// TODO: it might be nicer to pass these as an internal environment variable rather than
388388
// as Java options, due to complications with string parsing of nested quotes.
389-
javaOpts ++= Utils.sparkJavaOpts(sparkConf)
389+
// TODO: Use Utils.sparkJavaOpts here once we figure out how to deal with quotes and backslashes
390+
for ((k, v) <- sparkConf.getAll) {
391+
javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\""
392+
}
390393

391394
if (args.amClass == classOf[ApplicationMaster].getName) {
392395
sparkConf.getOption("spark.driver.extraJavaOptions")

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnableUtil.scala

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,13 @@ trait ExecutorRunnableUtil extends Logging {
6767
// registers with the Scheduler and transfers the spark configs. Since the Executor backend
6868
// uses Akka to connect to the scheduler, the akka settings are needed as well as the
6969
// authentication settings.
70-
javaOpts ++= Utils.sparkJavaOpts(sparkConf, SparkConf.isExecutorStartupConf)
70+
// TODO: Use Utils.sparkJavaOpts here once we figure out how to deal with quotes and backslashes
71+
sparkConf.getAll.
72+
filter { case (k, v) => k.startsWith("spark.auth") || k.startsWith("spark.akka") }.
73+
foreach { case (k, v) => javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\"" }
74+
75+
sparkConf.getAkkaConf.
76+
foreach { case (k, v) => javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\"" }
7177

7278
// Commenting it out for now - so that people can refer to the properties if required. Remove
7379
// it once cpuset version is pushed out.

0 commit comments

Comments
 (0)