Skip to content

Commit 287fa2f

Browse files
committed
- Updated documentation about specifying the hadoop version in building-spark. Now is clear that Spark will build against Hadoop 2.2.0 by default.
- Added Cloudera CDH 5.3.3 without MapReduce example in the building-spark doc.
1 parent 1354292 commit 287fa2f

File tree

1 file changed

+7
-3
lines changed

1 file changed

+7
-3
lines changed

docs/building-spark.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,16 +59,16 @@ You can fix this by setting the `MAVEN_OPTS` variable as discussed before.
5959

6060
# Specifying the Hadoop Version
6161

62-
Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the "hadoop.version" property. If unset, Spark will build against Hadoop 1.0.4 by default. Note that certain build profiles are required for particular Hadoop versions:
62+
Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the "hadoop.version" property. If unset, Spark will build against Hadoop 2.2.0 by default. Note that certain build profiles are required for particular Hadoop versions:
6363

6464
<table class="table">
6565
<thead>
6666
<tr><th>Hadoop version</th><th>Profile required</th></tr>
6767
</thead>
6868
<tbody>
6969
<tr><td>0.23.x</td><td>hadoop-0.23</td></tr>
70-
<tr><td>1.x to 2.1.x</td><td>(none)</td></tr>
71-
<tr><td>2.2.x</td><td>hadoop-2.2</td></tr>
70+
<tr><td>1.x to 2.1.x</td><td>hadoop-1</td></tr>
71+
<tr><td>2.2.x</td><td>(none)</td></tr>
7272
<tr><td>2.3.x</td><td>hadoop-2.3</td></tr>
7373
<tr><td>2.4.x</td><td>hadoop-2.4</td></tr>
7474
</tbody>
@@ -83,6 +83,10 @@ mvn -Dhadoop.version=1.2.1 -DskipTests clean package
8383
# Cloudera CDH 4.2.0 with MapReduce v1
8484
mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package
8585

86+
#Cloudera CDH 5.3.3 without MapReduce
87+
88+
mvn -Dhadoop.version=2.5.0-cdh5.3.3 -DskipTests clean package
89+
8690
# Apache Hadoop 0.23.x
8791
mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package
8892
{% endhighlight %}

0 commit comments

Comments
 (0)