Skip to content

Commit 88a2c20

Browse files
surongquansurongquan
committed
Merge branch 'master' of https://github.com/apache/spark into SPARK-3954
2 parents 178066f + 1ae51f6 commit 88a2c20

File tree

8,767 files changed

+67697
-55371
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

8,767 files changed

+67697
-55371
lines changed

‎.gitattributes

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*.bat text eol=crlf
2+
*.cmd text eol=crlf

‎.rat-excludes

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
target
22
.gitignore
3+
.gitattributes
34
.project
45
.classpath
56
.mima-excludes
@@ -48,6 +49,7 @@ sbt-launch-lib.bash
4849
plugins.sbt
4950
work
5051
.*\.q
52+
.*\.qv
5153
golden
5254
test.out/*
5355
.*iml

‎LICENSE

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -712,18 +712,6 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
712712
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
713713
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
714714

715-
========================================================================
716-
For colt:
717-
========================================================================
718-
719-
Copyright (c) 1999 CERN - European Organization for Nuclear Research.
720-
Permission to use, copy, modify, distribute and sell this software and its documentation for any purpose is hereby granted without fee, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation. CERN makes no representations about the suitability of this software for any purpose. It is provided "as is" without expressed or implied warranty.
721-
722-
Packages hep.aida.*
723-
724-
Written by Pavel Binko, Dino Ferrero Merlino, Wolfgang Hoschek, Tony Johnson, Andreas Pfeiffer, and others. Check the FreeHEP home page for more info. Permission to use and/or redistribute this work is granted under the terms of the LGPL License, with the exception that any usage related to military applications is expressly forbidden. The software and documentation made available under the terms of this license are provided with no warranty.
725-
726-
727715
========================================================================
728716
For SnapTree:
729717
========================================================================

‎README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ To build Spark and its example programs, run:
2525

2626
(You do not need to do this if you downloaded a pre-built package.)
2727
More detailed documentation is available from the project site, at
28-
["Building Spark"](http://spark.apache.org/docs/latest/building-spark.html).
28+
["Building Spark with Maven"](http://spark.apache.org/docs/latest/building-with-maven.html).
2929

3030
## Interactive Scala Shell
3131

@@ -84,7 +84,7 @@ storage systems. Because the protocols have changed in different versions of
8484
Hadoop, you must build Spark against the same version that your cluster runs.
8585

8686
Please refer to the build documentation at
87-
["Specifying the Hadoop Version"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version)
87+
["Specifying the Hadoop Version"](http://spark.apache.org/docs/latest/building-with-maven.html#specifying-the-hadoop-version)
8888
for detailed guidance on building for a particular distribution of Hadoop, including
8989
building for particular Hive and Hive Thriftserver distributions. See also
9090
["Third Party Hadoop Distributions"](http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html)

‎bin/compute-classpath.cmd

Lines changed: 117 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -1,117 +1,117 @@
1-
@echo off
2-
3-
rem
4-
rem Licensed to the Apache Software Foundation (ASF) under one or more
5-
rem contributor license agreements. See the NOTICE file distributed with
6-
rem this work for additional information regarding copyright ownership.
7-
rem The ASF licenses this file to You under the Apache License, Version 2.0
8-
rem (the "License"); you may not use this file except in compliance with
9-
rem the License. You may obtain a copy of the License at
10-
rem
11-
rem http://www.apache.org/licenses/LICENSE-2.0
12-
rem
13-
rem Unless required by applicable law or agreed to in writing, software
14-
rem distributed under the License is distributed on an "AS IS" BASIS,
15-
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16-
rem See the License for the specific language governing permissions and
17-
rem limitations under the License.
18-
rem
19-
20-
rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
21-
rem script and the ExecutorRunner in standalone cluster mode.
22-
23-
rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting
24-
rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we
25-
rem need to set it here because we use !datanucleus_jars! below.
26-
if "%DONT_PRINT_CLASSPATH%"=="1" goto skip_delayed_expansion
27-
setlocal enabledelayedexpansion
28-
:skip_delayed_expansion
29-
30-
set SCALA_VERSION=2.10
31-
32-
rem Figure out where the Spark framework is installed
33-
set FWDIR=%~dp0..\
34-
35-
rem Load environment variables from conf\spark-env.cmd, if it exists
36-
if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd"
37-
38-
rem Build up classpath
39-
set CLASSPATH=%SPARK_CLASSPATH%;%SPARK_SUBMIT_CLASSPATH%
40-
41-
if not "x%SPARK_CONF_DIR%"=="x" (
42-
set CLASSPATH=%CLASSPATH%;%SPARK_CONF_DIR%
43-
) else (
44-
set CLASSPATH=%CLASSPATH%;%FWDIR%conf
45-
)
46-
47-
if exist "%FWDIR%RELEASE" (
48-
for %%d in ("%FWDIR%lib\spark-assembly*.jar") do (
49-
set ASSEMBLY_JAR=%%d
50-
)
51-
) else (
52-
for %%d in ("%FWDIR%assembly\target\scala-%SCALA_VERSION%\spark-assembly*hadoop*.jar") do (
53-
set ASSEMBLY_JAR=%%d
54-
)
55-
)
56-
57-
set CLASSPATH=%CLASSPATH%;%ASSEMBLY_JAR%
58-
59-
rem When Hive support is needed, Datanucleus jars must be included on the classpath.
60-
rem Datanucleus jars do not work if only included in the uber jar as plugin.xml metadata is lost.
61-
rem Both sbt and maven will populate "lib_managed/jars/" with the datanucleus jars when Spark is
62-
rem built with Hive, so look for them there.
63-
if exist "%FWDIR%RELEASE" (
64-
set datanucleus_dir=%FWDIR%lib
65-
) else (
66-
set datanucleus_dir=%FWDIR%lib_managed\jars
67-
)
68-
set "datanucleus_jars="
69-
for %%d in ("%datanucleus_dir%\datanucleus-*.jar") do (
70-
set datanucleus_jars=!datanucleus_jars!;%%d
71-
)
72-
set CLASSPATH=%CLASSPATH%;%datanucleus_jars%
73-
74-
set SPARK_CLASSES=%FWDIR%core\target\scala-%SCALA_VERSION%\classes
75-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%repl\target\scala-%SCALA_VERSION%\classes
76-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%mllib\target\scala-%SCALA_VERSION%\classes
77-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%bagel\target\scala-%SCALA_VERSION%\classes
78-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%graphx\target\scala-%SCALA_VERSION%\classes
79-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%streaming\target\scala-%SCALA_VERSION%\classes
80-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%tools\target\scala-%SCALA_VERSION%\classes
81-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\classes
82-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\classes
83-
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\classes
84-
85-
set SPARK_TEST_CLASSES=%FWDIR%core\target\scala-%SCALA_VERSION%\test-classes
86-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%repl\target\scala-%SCALA_VERSION%\test-classes
87-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%mllib\target\scala-%SCALA_VERSION%\test-classes
88-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%bagel\target\scala-%SCALA_VERSION%\test-classes
89-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%graphx\target\scala-%SCALA_VERSION%\test-classes
90-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%streaming\target\scala-%SCALA_VERSION%\test-classes
91-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\test-classes
92-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\test-classes
93-
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\test-classes
94-
95-
if "x%SPARK_TESTING%"=="x1" (
96-
rem Add test clases to path - note, add SPARK_CLASSES and SPARK_TEST_CLASSES before CLASSPATH
97-
rem so that local compilation takes precedence over assembled jar
98-
set CLASSPATH=%SPARK_CLASSES%;%SPARK_TEST_CLASSES%;%CLASSPATH%
99-
)
100-
101-
rem Add hadoop conf dir - else FileSystem.*, etc fail
102-
rem Note, this assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
103-
rem the configurtion files.
104-
if "x%HADOOP_CONF_DIR%"=="x" goto no_hadoop_conf_dir
105-
set CLASSPATH=%CLASSPATH%;%HADOOP_CONF_DIR%
106-
:no_hadoop_conf_dir
107-
108-
if "x%YARN_CONF_DIR%"=="x" goto no_yarn_conf_dir
109-
set CLASSPATH=%CLASSPATH%;%YARN_CONF_DIR%
110-
:no_yarn_conf_dir
111-
112-
rem A bit of a hack to allow calling this script within run2.cmd without seeing output
113-
if "%DONT_PRINT_CLASSPATH%"=="1" goto exit
114-
115-
echo %CLASSPATH%
116-
117-
:exit
1+
@echo off
2+
3+
rem
4+
rem Licensed to the Apache Software Foundation (ASF) under one or more
5+
rem contributor license agreements. See the NOTICE file distributed with
6+
rem this work for additional information regarding copyright ownership.
7+
rem The ASF licenses this file to You under the Apache License, Version 2.0
8+
rem (the "License"); you may not use this file except in compliance with
9+
rem the License. You may obtain a copy of the License at
10+
rem
11+
rem http://www.apache.org/licenses/LICENSE-2.0
12+
rem
13+
rem Unless required by applicable law or agreed to in writing, software
14+
rem distributed under the License is distributed on an "AS IS" BASIS,
15+
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
rem See the License for the specific language governing permissions and
17+
rem limitations under the License.
18+
rem
19+
20+
rem This script computes Spark's classpath and prints it to stdout; it's used by both the "run"
21+
rem script and the ExecutorRunner in standalone cluster mode.
22+
23+
rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting
24+
rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we
25+
rem need to set it here because we use !datanucleus_jars! below.
26+
if "%DONT_PRINT_CLASSPATH%"=="1" goto skip_delayed_expansion
27+
setlocal enabledelayedexpansion
28+
:skip_delayed_expansion
29+
30+
set SCALA_VERSION=2.10
31+
32+
rem Figure out where the Spark framework is installed
33+
set FWDIR=%~dp0..\
34+
35+
rem Load environment variables from conf\spark-env.cmd, if it exists
36+
if exist "%FWDIR%conf\spark-env.cmd" call "%FWDIR%conf\spark-env.cmd"
37+
38+
rem Build up classpath
39+
set CLASSPATH=%SPARK_CLASSPATH%;%SPARK_SUBMIT_CLASSPATH%
40+
41+
if not "x%SPARK_CONF_DIR%"=="x" (
42+
set CLASSPATH=%CLASSPATH%;%SPARK_CONF_DIR%
43+
) else (
44+
set CLASSPATH=%CLASSPATH%;%FWDIR%conf
45+
)
46+
47+
if exist "%FWDIR%RELEASE" (
48+
for %%d in ("%FWDIR%lib\spark-assembly*.jar") do (
49+
set ASSEMBLY_JAR=%%d
50+
)
51+
) else (
52+
for %%d in ("%FWDIR%assembly\target\scala-%SCALA_VERSION%\spark-assembly*hadoop*.jar") do (
53+
set ASSEMBLY_JAR=%%d
54+
)
55+
)
56+
57+
set CLASSPATH=%CLASSPATH%;%ASSEMBLY_JAR%
58+
59+
rem When Hive support is needed, Datanucleus jars must be included on the classpath.
60+
rem Datanucleus jars do not work if only included in the uber jar as plugin.xml metadata is lost.
61+
rem Both sbt and maven will populate "lib_managed/jars/" with the datanucleus jars when Spark is
62+
rem built with Hive, so look for them there.
63+
if exist "%FWDIR%RELEASE" (
64+
set datanucleus_dir=%FWDIR%lib
65+
) else (
66+
set datanucleus_dir=%FWDIR%lib_managed\jars
67+
)
68+
set "datanucleus_jars="
69+
for %%d in ("%datanucleus_dir%\datanucleus-*.jar") do (
70+
set datanucleus_jars=!datanucleus_jars!;%%d
71+
)
72+
set CLASSPATH=%CLASSPATH%;%datanucleus_jars%
73+
74+
set SPARK_CLASSES=%FWDIR%core\target\scala-%SCALA_VERSION%\classes
75+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%repl\target\scala-%SCALA_VERSION%\classes
76+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%mllib\target\scala-%SCALA_VERSION%\classes
77+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%bagel\target\scala-%SCALA_VERSION%\classes
78+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%graphx\target\scala-%SCALA_VERSION%\classes
79+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%streaming\target\scala-%SCALA_VERSION%\classes
80+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%tools\target\scala-%SCALA_VERSION%\classes
81+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\classes
82+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\classes
83+
set SPARK_CLASSES=%SPARK_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\classes
84+
85+
set SPARK_TEST_CLASSES=%FWDIR%core\target\scala-%SCALA_VERSION%\test-classes
86+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%repl\target\scala-%SCALA_VERSION%\test-classes
87+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%mllib\target\scala-%SCALA_VERSION%\test-classes
88+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%bagel\target\scala-%SCALA_VERSION%\test-classes
89+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%graphx\target\scala-%SCALA_VERSION%\test-classes
90+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%streaming\target\scala-%SCALA_VERSION%\test-classes
91+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\catalyst\target\scala-%SCALA_VERSION%\test-classes
92+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\core\target\scala-%SCALA_VERSION%\test-classes
93+
set SPARK_TEST_CLASSES=%SPARK_TEST_CLASSES%;%FWDIR%sql\hive\target\scala-%SCALA_VERSION%\test-classes
94+
95+
if "x%SPARK_TESTING%"=="x1" (
96+
rem Add test clases to path - note, add SPARK_CLASSES and SPARK_TEST_CLASSES before CLASSPATH
97+
rem so that local compilation takes precedence over assembled jar
98+
set CLASSPATH=%SPARK_CLASSES%;%SPARK_TEST_CLASSES%;%CLASSPATH%
99+
)
100+
101+
rem Add hadoop conf dir - else FileSystem.*, etc fail
102+
rem Note, this assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
103+
rem the configurtion files.
104+
if "x%HADOOP_CONF_DIR%"=="x" goto no_hadoop_conf_dir
105+
set CLASSPATH=%CLASSPATH%;%HADOOP_CONF_DIR%
106+
:no_hadoop_conf_dir
107+
108+
if "x%YARN_CONF_DIR%"=="x" goto no_yarn_conf_dir
109+
set CLASSPATH=%CLASSPATH%;%YARN_CONF_DIR%
110+
:no_yarn_conf_dir
111+
112+
rem A bit of a hack to allow calling this script within run2.cmd without seeing output
113+
if "%DONT_PRINT_CLASSPATH%"=="1" goto exit
114+
115+
echo %CLASSPATH%
116+
117+
:exit

‎bin/pyspark2.cmd

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,12 @@ for /f %%i in ('echo %1^| findstr /R "\.py"') do (
5959
)
6060

6161
if [%PYTHON_FILE%] == [] (
62-
%PYSPARK_PYTHON%
62+
set PYSPARK_SHELL=1
63+
if [%IPYTHON%] == [1] (
64+
ipython %IPYTHON_OPTS%
65+
) else (
66+
%PYSPARK_PYTHON%
67+
)
6368
) else (
6469
echo.
6570
echo WARNING: Running python applications through ./bin/pyspark.cmd is deprecated as of Spark 1.0.

‎bin/spark-class

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,11 @@ case "$1" in
8181
OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_SUBMIT_OPTS"
8282
OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM}
8383
if [ -n "$SPARK_SUBMIT_LIBRARY_PATH" ]; then
84-
OUR_JAVA_OPTS="$OUR_JAVA_OPTS -Djava.library.path=$SPARK_SUBMIT_LIBRARY_PATH"
84+
if [[ $OSTYPE == darwin* ]]; then
85+
export DYLD_LIBRARY_PATH="$SPARK_SUBMIT_LIBRARY_PATH:$DYLD_LIBRARY_PATH"
86+
else
87+
export LD_LIBRARY_PATH="$SPARK_SUBMIT_LIBRARY_PATH:$LD_LIBRARY_PATH"
88+
fi
8589
fi
8690
if [ -n "$SPARK_SUBMIT_DRIVER_MEMORY" ]; then
8791
OUR_JAVA_MEM="$SPARK_SUBMIT_DRIVER_MEMORY"

‎core/pom.xml

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,16 @@
4444
</exclusion>
4545
</exclusions>
4646
</dependency>
47+
<dependency>
48+
<groupId>org.apache.spark</groupId>
49+
<artifactId>spark-network-common_2.10</artifactId>
50+
<version>${project.version}</version>
51+
</dependency>
52+
<dependency>
53+
<groupId>org.apache.spark</groupId>
54+
<artifactId>spark-network-shuffle_2.10</artifactId>
55+
<version>${project.version}</version>
56+
</dependency>
4757
<dependency>
4858
<groupId>net.java.dev.jets3t</groupId>
4959
<artifactId>jets3t</artifactId>
@@ -85,8 +95,6 @@
8595
<dependency>
8696
<groupId>org.apache.commons</groupId>
8797
<artifactId>commons-math3</artifactId>
88-
<version>3.3</version>
89-
<scope>test</scope>
9098
</dependency>
9199
<dependency>
92100
<groupId>com.google.code.findbugs</groupId>
@@ -132,6 +140,10 @@
132140
<groupId>com.twitter</groupId>
133141
<artifactId>chill-java</artifactId>
134142
</dependency>
143+
<dependency>
144+
<groupId>org.roaringbitmap</groupId>
145+
<artifactId>RoaringBitmap</artifactId>
146+
</dependency>
135147
<dependency>
136148
<groupId>commons-net</groupId>
137149
<artifactId>commons-net</artifactId>
@@ -158,10 +170,6 @@
158170
<artifactId>json4s-jackson_${scala.binary.version}</artifactId>
159171
<version>3.2.10</version>
160172
</dependency>
161-
<dependency>
162-
<groupId>colt</groupId>
163-
<artifactId>colt</artifactId>
164-
</dependency>
165173
<dependency>
166174
<groupId>org.apache.mesos</groupId>
167175
<artifactId>mesos</artifactId>
@@ -243,6 +251,11 @@
243251
</exclusion>
244252
</exclusions>
245253
</dependency>
254+
<dependency>
255+
<groupId>org.seleniumhq.selenium</groupId>
256+
<artifactId>selenium-java</artifactId>
257+
<scope>test</scope>
258+
</dependency>
246259
<dependency>
247260
<groupId>org.scalatest</groupId>
248261
<artifactId>scalatest_${scala.binary.version}</artifactId>

0 commit comments

Comments
 (0)