Skip to content

Commit d4df076

Browse files
tgravescsmateiz
authored andcommitted
Update documentation for work around for SPARK-1384
This is to workaround accessing secure hdfs from spark-shell in yarn-client mode. Note this only applies to branch-0.9 and is intended to be included in the documentation for 0.9.1. The real fix after 0.9.1 is included in #287 Author: Thomas Graves <[email protected]> Closes #314 from tgravescs/docFix09rc3 and squashes the following commits: 222e848 [Thomas Graves] Update documentation for work around for SPARK-1384
1 parent 7f727cf commit d4df076

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

docs/running-on-yarn.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ With this mode, your application is actually run on the remote machine where the
9999

100100
## Launch spark application with yarn-client mode.
101101

102-
With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR
102+
With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR. If you are using spark-shell with secure HDFS you also need to export SPARK_YARN_MODE=true.
103103

104104
Configuration in yarn-client mode:
105105

@@ -121,7 +121,7 @@ For example:
121121
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
122122
./bin/run-example org.apache.spark.examples.SparkPi yarn-client
123123

124-
124+
SPARK_YARN_MODE=true \
125125
SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \
126126
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
127127
MASTER=yarn-client ./bin/spark-shell

0 commit comments

Comments
 (0)