Skip to content

Commit ace454d

Browse files
committed
SPARK-7265 take out PySpark on YARN limitation
1 parent 9ea0832 commit ace454d

File tree

1 file changed

+0
-1
lines changed

1 file changed

+0
-1
lines changed

docs/running-on-yarn.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -305,4 +305,3 @@ If you need a reference to the proper location to put log files in the YARN so t
305305
- In `yarn-cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `yarn-client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `yarn-client` mode, only the Spark executors do.
306306
- The `--files` and `--archives` options support specifying file names with the # similar to Hadoop. For example you can specify: `--files localtest.txt#appSees.txt` and this will upload the file you have locally named localtest.txt into HDFS but this will be linked to by the name `appSees.txt`, and your application should use the name as `appSees.txt` to reference it when running on YARN.
307307
- The `--jars` option allows the `SparkContext.addJar` function to work if you are using it with local files and running in `yarn-cluster` mode. It does not need to be used if you are using it with HDFS, HTTP, HTTPS, or FTP files.
308-
- In order to make PySpark work on YARN, please build spark with Java 6.

0 commit comments

Comments
 (0)