You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/running-on-yarn.md
+5-5Lines changed: 5 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -211,18 +211,18 @@ settings and a restart of all node managers. Thus, this is not applicable to hos
211
211
212
212
To use a custom log4j configuration for the application master or executors, there are two options:
213
213
214
-
- upload a custom log4j.properties using spark-submit, by adding it to the "--files" list of files
214
+
- upload a custom `log4j.properties` using `spark-submit`, by adding it to the `--files` list of files
215
215
to be uploaded with the application.
216
-
- add "-Dlog4j.configuration=<locationofconfigurationfile>" to "spark.driver.extraJavaOptions"
217
-
(for the driver) or "spark.executor.extraJavaOptions" (for executors). Note that if using a file,
218
-
the "file:" protocol should be explicitly provided, and the file needs to exist locally on all
216
+
- add `-Dlog4j.configuration=<location of configuration file>` to `spark.driver.extraJavaOptions`
217
+
(for the driver) or `spark.executor.extraJavaOptions` (for executors). Note that if using a file,
218
+
the `file:` protocol should be explicitly provided, and the file needs to exist locally on all
219
219
the nodes.
220
220
221
221
Note that for the first option, both executors and the application master will share the same
222
222
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
223
223
to the same log file).
224
224
225
-
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use "${spark.yarn.app.container.log.dir}" in your log4j.properties. For example, log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log. For streaming application, configuring RollingFileAppender and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.
225
+
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use `spark.yarn.app.container.log.dir` in your log4j.properties. For example, `log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log`. For streaming application, configuring `RollingFileAppender` and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.
0 commit comments