Skip to content

[SPARK-8776] Increase the default MaxPermSize #7196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

[SPARK-8776] Increase the default MaxPermSize #7196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
30aaf8e
Increase the default PermGen size to 256m.
yhuai Jul 2, 2015
d44b713
Make sparkShell and hiveConsole use 256m PermGen size.
yhuai Jul 2, 2015
60901b4
Fix test.
yhuai Jul 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ void addPermGenSizeOpt(List<String> cmd) {
}
}

cmd.add("-XX:MaxPermSize=128m");
cmd.add("-XX:MaxPermSize=256m");
}

void addOptionString(List<String> cmd, String options) {
Expand Down
2 changes: 1 addition & 1 deletion launcher/src/test/java/org/apache/spark/launcher/SparkSubmitCommandBuilderSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ private void testCmdBuilder(boolean isDriver) throws Exception {
if (isDriver) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove this condition then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, yes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not worth another PR since it's already merged

assertEquals("-XX:MaxPermSize=256m", arg);
} else {
assertEquals("-XX:MaxPermSize=128m", arg);
assertEquals("-XX:MaxPermSize=256m", arg);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewor14 What does isDriver do at here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it literally just says give more perm space for the driver than executors. This change is probably OK.

}
}
}
Expand Down
4 changes: 2 additions & 2 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ object SparkBuild extends PomBuild {
fork := true,
outputStrategy in run := Some (StdoutOutput),

javaOptions ++= Seq("-Xmx2G", "-XX:MaxPermSize=1g"),
javaOptions ++= Seq("-Xmx2G", "-XX:MaxPermSize=256m"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marmbrus This setting only affect sparkShell and sparkSql, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and probably the tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait, why do we want to lower it here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just want to align it with what happens when you run the spark-shell normally.


sparkShell := {
(runMain in Compile).toTask(" org.apache.spark.repl.Main -usejavacp").value
Expand Down Expand Up @@ -299,7 +299,7 @@ object SQL {
object Hive {

lazy val settings = Seq(
javaOptions += "-XX:MaxPermSize=1g",
javaOptions += "-XX:MaxPermSize=256m",
// Specially disable assertions since some Hive tests fail them
javaOptions in Test := (javaOptions in Test).value.filterNot(_ == "-ea"),
// Multiple queries rely on the TestHive singleton. See comments there for more details.
Expand Down