[SPARK-2098] All Spark processes should support spark-defaults.conf, config file #1256

witgo · 2014-06-28T14:15:23Z

No description provided.

AmplabJenkins · 2014-06-28T14:15:30Z

Merged build triggered.

AmplabJenkins · 2014-06-28T14:15:38Z

Merged build started.

AmplabJenkins · 2014-06-28T14:25:30Z

Merged build triggered.

AmplabJenkins · 2014-06-28T14:25:38Z

Merged build started.

AmplabJenkins · 2014-06-28T14:59:40Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-28T14:59:40Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16228/

AmplabJenkins · 2014-06-28T15:08:24Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-28T15:08:24Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16229/

AmplabJenkins · 2014-06-30T16:00:43Z

Merged build triggered.

AmplabJenkins · 2014-06-30T16:00:50Z

Merged build started.

AmplabJenkins · 2014-06-30T16:45:55Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-30T16:45:55Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16267/

vanzin · 2014-07-08T17:19:51Z

core/src/main/scala/org/apache/spark/SparkConf.scala

+  }
+
+  /** Load any spark.* system properties */
+  private[spark] def loadSystemProperties(isOverride: Boolean = true) {


This is only called in one place, why the argument? Seems like we always want system properties to override other settings anyway.

vanzin · 2014-07-08T17:47:02Z

I'd like to see better documentation about the semantics of how the different sources for configuration override each other. Also, some tests would be nice.

AmplabJenkins · 2014-07-09T15:31:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T15:31:17Z

Merged build started.

AmplabJenkins · 2014-07-09T15:41:21Z

Merged build finished.

AmplabJenkins · 2014-07-09T15:41:22Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16465/

witgo · 2014-07-09T15:45:58Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-09T15:46:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T15:46:17Z

Merged build started.

vanzin · 2014-07-09T17:37:54Z

core/src/main/scala/org/apache/spark/util/Utils.scala

+        val message = s"Failed when loading Spark properties"
+        throw new SparkException(message, e)
+    } finally {
+      inReader.close()


If you're keeping this, you should document that this method will close the given input stream.

vanzin · 2014-07-30T17:13:16Z

core/src/test/scala/org/apache/spark/SparkConfSuite.scala

+  test("loading from file") {
+    val outFile = File.createTempFile("sparkConf-loading-from-file", "")
+    outFile.deleteOnExit()
+    val outStream: FileWriter = new FileWriter(outFile)


You could use this method instead.

vanzin · 2014-07-30T17:15:07Z

Still a few nits left, but looks ok to me.

SparkQA · 2014-07-31T02:28:59Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17539/consoleFull

SparkQA · 2014-07-31T03:23:49Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17539/consoleFull

SparkQA · 2014-07-31T05:48:57Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17555/consoleFull

SparkQA · 2014-07-31T06:28:49Z

QA results for PR 1256:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17555/consoleFull

witgo · 2014-07-31T06:32:16Z

Jenkins, retest this please.

SparkQA · 2014-07-31T06:34:07Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17564/consoleFull

SparkQA · 2014-07-31T07:30:03Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17564/consoleFull

SparkQA · 2014-08-11T06:39:43Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18302/consoleFull

SparkQA · 2014-08-11T07:30:12Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18302/consoleFull

vanzin · 2014-08-22T00:54:30Z

@pwendell @andrewor14 could you guys take a look at this PR? Thanks!

andrewor14 · 2014-09-11T23:53:02Z

docs/monitoring.md

+    <td>spark.history.fs.logDirectory</td>
+    <td>(none)</td>
+    <td>
+      Directory where app logs are stored.


How about "Directory that contains application event logs to be loaded by the history server"

andrewor14 · 2014-09-12T00:14:44Z

@witgo I like the idea of having all processes (HistoryServer, Master, Worker etc.) read from the properties file in addition just SparkSubmit. However, I don't think this scope should include SparkConf, which is more application-specific. It would be good to keep the existing code in SparkSubmitArguments and only move the part where we actually read properties from the file into Utils, then all the other processes can handle --properties-file the same way. Do you understand what I mean?

witgo · 2014-09-12T02:13:40Z

OK, I generally understand what you mean,I will re-implement the feature at the weekend.

andrewor14 · 2014-09-13T08:00:35Z

@witgo would you mind closing this since you opened another one?

witgo · 2014-09-13T08:02:48Z

OK

…config file This is another implementation about #1256 cc andrewor14 vanzin Author: GuoQiang Li <[email protected]> Closes #2379 from witgo/SPARK-2098-new and squashes the following commits: 4ef1cbd [GuoQiang Li] review commit 49ef70e [GuoQiang Li] Refactor getDefaultPropertiesFile c45d20c [GuoQiang Li] All Spark processes should support spark-defaults.conf, config file

* [CARMEL-6541] Support Query Level SQL Conf leveraging Hint * fix code style * fix ut * Add one more test Co-authored-by: Chao Sun <[email protected]>

witgo mentioned this pull request Jun 28, 2014

[WIP] Loading spark-defaults.conf when creating SparkConf instances #1233

Closed

vanzin reviewed Jul 8, 2014
View reviewed changes

vanzin reviewed Jul 9, 2014
View reviewed changes

vanzin reviewed Jul 30, 2014
View reviewed changes

All Spark processes should support spark-defaults.conf, config file.

62f5677

andrewor14 reviewed Sep 11, 2014
View reviewed changes

witgo changed the title ~~SPARK-2098: All Spark processes should support spark-defaults.conf, config file~~ [SPARK-2098] All Spark processes should support spark-defaults.conf, config file Sep 13, 2014

witgo mentioned this pull request Sep 13, 2014

[SPARK-2098] All Spark processes should support spark-defaults.conf, config file #2379

Closed

witgo closed this Sep 13, 2014

witgo deleted the SPARK-2098 branch October 15, 2014 10:22

[SPARK-2098] All Spark processes should support spark-defaults.conf, config file #1256

[SPARK-2098] All Spark processes should support spark-defaults.conf, config file #1256

Uh oh!

Conversation

witgo commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

vanzin Jul 8, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin commented Jul 8, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

witgo commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

vanzin Jul 9, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin Jul 30, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin commented Jul 30, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

witgo commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

vanzin commented Aug 22, 2014

Uh oh!

andrewor14 Sep 11, 2014

Choose a reason for hiding this comment

Uh oh!