[SPARK-8776] Increase the default MaxPermSize #7196

yhuai · 2015-07-02T20:00:40Z

I am increasing the perm gen size to 256m.

https://issues.apache.org/jira/browse/SPARK-8776

AmplabJenkins · 2015-07-02T20:03:11Z

Merged build triggered.

AmplabJenkins · 2015-07-02T20:03:20Z

Merged build started.

SparkQA · 2015-07-02T20:06:59Z

Test build #36426 has started for PR 7196 at commit 30aaf8e.

AmplabJenkins · 2015-07-02T20:23:11Z

Merged build triggered.

AmplabJenkins · 2015-07-02T20:23:20Z

Merged build started.

SparkQA · 2015-07-02T20:26:37Z

Test build #36428 has started for PR 7196 at commit d44b713.

yhuai · 2015-07-02T20:30:02Z

project/SparkBuild.scala

@@ -206,7 +206,7 @@ object SparkBuild extends PomBuild {
    fork := true,
    outputStrategy in run := Some (StdoutOutput),

-    javaOptions ++= Seq("-Xmx2G", "-XX:MaxPermSize=1g"),
+    javaOptions ++= Seq("-Xmx2G", "-XX:MaxPermSize=256m"),


@marmbrus This setting only affect sparkShell and sparkSql, right?

and probably the tests

wait, why do we want to lower it here?

We just want to align it with what happens when you run the spark-shell normally.

SparkQA · 2015-07-02T22:02:11Z

Test build #36426 has finished for PR 7196 at commit 30aaf8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateNamedStruct(children: Seq[Expression]) extends Expression

AmplabJenkins · 2015-07-02T22:02:25Z

Merged build finished. Test FAILed.

yhuai · 2015-07-02T22:14:40Z

test this please.

AmplabJenkins · 2015-07-02T22:18:10Z

Merged build triggered.

AmplabJenkins · 2015-07-02T22:18:16Z

Merged build started.

SparkQA · 2015-07-02T22:21:38Z

Test build #36438 has started for PR 7196 at commit d44b713.

SparkQA · 2015-07-02T22:30:34Z

Test build #36428 has finished for PR 7196 at commit d44b713.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateNamedStruct(children: Seq[Expression]) extends Expression
- case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
- case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression

AmplabJenkins · 2015-07-02T22:30:50Z

Merged build finished. Test FAILed.

SparkQA · 2015-07-03T00:19:11Z

Test build #36438 has finished for PR 7196 at commit d44b713.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-03T00:19:26Z

Merged build finished. Test FAILed.

andrewor14 · 2015-07-03T01:49:48Z

retest this please

AmplabJenkins · 2015-07-03T01:53:12Z

Merged build triggered.

AmplabJenkins · 2015-07-03T01:53:21Z

Merged build started.

SparkQA · 2015-07-03T01:54:42Z

Test build #36460 has started for PR 7196 at commit d44b713.

yhuai · 2015-07-03T02:50:06Z

launcher/src/test/java/org/apache/spark/launcher/SparkSubmitCommandBuilderSuite.java

@@ -194,7 +194,7 @@ private void testCmdBuilder(boolean isDriver) throws Exception {
        if (isDriver) {
          assertEquals("-XX:MaxPermSize=256m", arg);
        } else {
-          assertEquals("-XX:MaxPermSize=128m", arg);
+          assertEquals("-XX:MaxPermSize=256m", arg);


@andrewor14 What does isDriver do at here?

I think it literally just says give more perm space for the driver than executors. This change is probably OK.

AmplabJenkins · 2015-07-03T02:53:10Z

Merged build triggered.

AmplabJenkins · 2015-07-03T02:53:16Z

Merged build started.

SparkQA · 2015-07-03T02:56:15Z

Test build #36467 has started for PR 7196 at commit 60901b4.

SparkQA · 2015-07-03T04:23:15Z

Test build #36460 has finished for PR 7196 at commit d44b713.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-03T04:23:30Z

Merged build finished. Test FAILed.

SparkQA · 2015-07-03T05:05:14Z

Test build #36467 has finished for PR 7196 at commit 60901b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-03T05:05:51Z

Merged build finished. Test PASSed.

yhuai · 2015-07-03T05:08:06Z

OK. I am merging it to master and branch 1.4.

I am increasing the perm gen size to 256m. https://issues.apache.org/jira/browse/SPARK-8776 Author: Yin Huai <[email protected]> Closes #7196 from yhuai/SPARK-8776 and squashes the following commits: 60901b4 [Yin Huai] Fix test. d44b713 [Yin Huai] Make sparkShell and hiveConsole use 256m PermGen size. 30aaf8e [Yin Huai] Increase the default PermGen size to 256m. (cherry picked from commit f743c79) Signed-off-by: Yin Huai <[email protected]>

srowen · 2015-07-03T06:30:47Z

launcher/src/test/java/org/apache/spark/launcher/SparkSubmitCommandBuilderSuite.java

@@ -194,7 +194,7 @@ private void testCmdBuilder(boolean isDriver) throws Exception {
        if (isDriver) {


You can remove this condition then?

Probably not worth another PR since it's already merged

Increase the default PermGen size to 256m.

30aaf8e

Make sparkShell and hiveConsole use 256m PermGen size.

d44b713

yhuai reviewed Jul 2, 2015
View reviewed changes

Fix test.

60901b4

yhuai reviewed Jul 3, 2015
View reviewed changes

asfgit closed this in f743c79 Jul 3, 2015

srowen reviewed Jul 3, 2015
View reviewed changes

		@@ -194,7 +194,7 @@ private void testCmdBuilder(boolean isDriver) throws Exception {
		if (isDriver) {

[SPARK-8776] Increase the default MaxPermSize #7196

[SPARK-8776] Increase the default MaxPermSize #7196

Uh oh!

Conversation

yhuai commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

SparkQA commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

SparkQA commented Jul 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

yhuai commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

SparkQA commented Jul 2, 2015

Uh oh!

SparkQA commented Jul 2, 2015

Uh oh!

AmplabJenkins commented Jul 2, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

andrewor14 commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

SparkQA commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

yhuai commented Jul 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!