Skip to content

Commit 38e624a

Browse files
author
Andrew Or
committed
[SPARK-5816] Add huge compatibility warning in DriverWrapper
The stability of the new submission gateway assumes that the arguments in `DriverWrapper` are consistent across multiple Spark versions. However, this is not at all clear from the code itself. In fact, this was broken in 20a6013, which is fortunately OK because both that commit and the original commit that added this gateway are part of the same release. To prevent this from happening again we should at the very least add a huge warning where appropriate. Author: Andrew Or <[email protected]> Closes #4687 from andrewor14/driver-wrapper-warning and squashes the following commits: 7989b56 [Andrew Or] Add huge compatibility warning
1 parent fb87f44 commit 38e624a

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,17 @@ import org.apache.spark.util.{AkkaUtils, ChildFirstURLClassLoader, MutableURLCla
2626

2727
/**
2828
* Utility object for launching driver programs such that they share fate with the Worker process.
29+
* This is used in standalone cluster mode only.
2930
*/
3031
object DriverWrapper {
3132
def main(args: Array[String]) {
3233
args.toList match {
34+
/*
35+
* IMPORTANT: Spark 1.3 provides a stable application submission gateway that is both
36+
* backward and forward compatible across future Spark versions. Because this gateway
37+
* uses this class to launch the driver, the ordering and semantics of the arguments
38+
* here must also remain consistent across versions.
39+
*/
3340
case workerUrl :: userJar :: mainClass :: extraArgs =>
3441
val conf = new SparkConf()
3542
val (actorSystem, _) = AkkaUtils.createActorSystem("Driver",

0 commit comments

Comments
 (0)