You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/building-spark.md
+11-12Lines changed: 11 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -61,12 +61,13 @@ If you don't run this, you may see errors like the following:
61
61
You can fix this by setting the `MAVEN_OPTS` variable as discussed before.
62
62
63
63
**Note:**
64
-
**For Java 8 and above this step is not required.*
65
-
**If using `build/mvn` and `MAVEN_OPTS` were not already set, the script will automate this for you.*
64
+
65
+
* For Java 8 and above this step is not required.
66
+
* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automate this for you.
66
67
67
68
# Specifying the Hadoop Version
68
69
69
-
Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the "hadoop.version" property. If unset, Spark will build against Hadoop 2.2.0 by default. Note that certain build profiles are required for particular Hadoop versions:
70
+
Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the `hadoop.version` property. If unset, Spark will build against Hadoop 2.2.0 by default. Note that certain build profiles are required for particular Hadoop versions:
You can enable the "yarn" profile and optionally set the "yarn.version" property if it is different from "hadoop.version". Spark only supports YARN versions 2.2.0 and later.
95
+
You can enable the `yarn` profile and optionally set the `yarn.version` property if it is different from `hadoop.version`. Spark only supports YARN versions 2.2.0 and later.
Spark does not yet support its JDBC component for Scala 2.11.
@@ -163,11 +164,9 @@ the `spark-parent` module).
163
164
164
165
Thus, the full flow for running continuous-compilation of the `core` submodule may look more like:
165
166
166
-
```
167
-
$ mvn install
168
-
$ cd core
169
-
$ mvn scala:cc
170
-
```
167
+
$ mvn install
168
+
$ cd core
169
+
$ mvn scala:cc
171
170
172
171
# Building Spark with IntelliJ IDEA or Eclipse
173
172
@@ -193,11 +192,11 @@ then ship it over to the cluster. We are investigating the exact cause for this.
193
192
194
193
# Packaging without Hadoop Dependencies for YARN
195
194
196
-
The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
195
+
The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with `yarn.application.classpath`. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
197
196
198
197
# Building with SBT
199
198
200
-
Maven is the official recommendation for packaging Spark, and is the "build of reference".
199
+
Maven is the official build tool recommended for packaging Spark, and is the *build of reference*.
201
200
But SBT is supported for day-to-day development since it can provide much faster iterative
202
201
compilation. More advanced developers may wish to use SBT.
0 commit comments