[SPARK-5931][CORE] Use consistent naming for time properties #5236

ilganeli · 2015-03-27T21:17:04Z

I've added new utility methods to do the conversion from times specified as e.g. 120s, 240ms, 360us to convert to a consistent internal representation. I've updated usage of these constants throughout the code to be consistent.

I believe I've captured all usages of time-based properties throughout the code. I've also updated variable names in a number of places to reflect their units for clarity and updated documentation where appropriate.

…s to handle time strings. Updated time strings in a few places to properly parse time

SparkQA · 2015-03-27T22:44:17Z

Test build #29325 has finished for PR 5236 at commit 59bf9e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer]

SparkQA · 2015-03-27T22:48:59Z

Test build #29329 has finished for PR 5236 at commit cde9bff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer]

SparkQA · 2015-03-27T22:51:57Z

Test build #29330 has finished for PR 5236 at commit 42477aa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer]

SparkQA · 2015-03-27T23:08:51Z

Test build #29326 has finished for PR 5236 at commit 4933fda.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer]

SparkQA · 2015-03-27T23:09:14Z

Test build #29328 has finished for PR 5236 at commit 064ebd6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer]

ilganeli · 2015-03-27T23:18:21Z

retest this please

SparkQA · 2015-03-27T23:42:38Z

Test build #29331 has finished for PR 5236 at commit 9a29d8d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Fixed ordering to fix overlap of s with ms and us.

Fixed overlap of (s) with (ms) and (us) in cascading if.

andrewor14 · 2015-04-13T17:50:55Z

core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala

+      case TaskLocality.PROCESS_LOCAL => "spark.locality.wait.process"
+      case TaskLocality.NODE_LOCAL => "spark.locality.wait.node"
+      case TaskLocality.RACK_LOCAL => "spark.locality.wait.rack"
+      case _ => ""


this should be return 0L instead to match old behavior (probably a corner case)

SparkQA · 2015-04-13T18:30:09Z

Test build #30176 has finished for PR 5236 at commit 951ca2d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

andrewor14 · 2015-04-13T18:32:37Z

docs/configuration.md

+    5d (days)
+    1y (years)
+
+If no units are provided, the default unit specified in the documentation is assumed. 


[Edit] Actually, scratch my last comment. This is probably fine.

SparkQA · 2015-04-13T18:44:56Z

Test build #30179 has finished for PR 5236 at commit f5fafcd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

andrewor14 · 2015-04-13T18:50:49Z

I just took a quick look at the existing configs that have units. It appears that none of them are documented, fortunately. I think this patch is more or less ready for merge once it passes tests.

SparkQA · 2015-04-13T20:44:23Z

Test build #30184 has finished for PR 5236 at commit de3bff9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:
- RoaringBitmap-0.4.5.jar
- activation-1.1.jar
- akka-actor_2.10-2.3.4-spark.jar
- akka-remote_2.10-2.3.4-spark.jar
- akka-slf4j_2.10-2.3.4-spark.jar
- aopalliance-1.0.jar
- arpack_combined_all-0.1.jar
- avro-1.7.7.jar
- breeze-macros_2.10-0.11.2.jar
- breeze_2.10-0.11.2.jar
- chill-java-0.5.0.jar
- chill_2.10-0.5.0.jar
- commons-beanutils-1.7.0.jar
- commons-beanutils-core-1.8.0.jar
- commons-cli-1.2.jar
- commons-codec-1.10.jar
- commons-collections-3.2.1.jar
- commons-compress-1.4.1.jar
- commons-configuration-1.6.jar
- commons-digester-1.8.jar
- commons-httpclient-3.1.jar
- commons-io-2.1.jar
- commons-lang-2.5.jar
- commons-lang3-3.3.2.jar
- commons-math-2.1.jar
- commons-math3-3.1.1.jar
- commons-net-2.2.jar
- compress-lzf-1.0.0.jar
- config-1.2.1.jar
- core-1.1.2.jar
- curator-client-2.4.0.jar
- curator-framework-2.4.0.jar
- curator-recipes-2.4.0.jar
- gmbal-api-only-3.0.0-b023.jar
- grizzly-framework-2.1.2.jar
- grizzly-http-2.1.2.jar
- grizzly-http-server-2.1.2.jar
- grizzly-http-servlet-2.1.2.jar
- grizzly-rcm-2.1.2.jar
- groovy-all-2.3.7.jar
- guava-14.0.1.jar
- guice-3.0.jar
- hadoop-annotations-2.2.0.jar
- hadoop-auth-2.2.0.jar
- hadoop-client-2.2.0.jar
- hadoop-common-2.2.0.jar
- hadoop-hdfs-2.2.0.jar
- hadoop-mapreduce-client-app-2.2.0.jar
- hadoop-mapreduce-client-common-2.2.0.jar
- hadoop-mapreduce-client-core-2.2.0.jar
- hadoop-mapreduce-client-jobclient-2.2.0.jar
- hadoop-mapreduce-client-shuffle-2.2.0.jar
- hadoop-yarn-api-2.2.0.jar
- hadoop-yarn-client-2.2.0.jar
- hadoop-yarn-common-2.2.0.jar
- hadoop-yarn-server-common-2.2.0.jar
- ivy-2.4.0.jar
- jackson-annotations-2.4.0.jar
- jackson-core-2.4.4.jar
- jackson-core-asl-1.8.8.jar
- jackson-databind-2.4.4.jar
- jackson-jaxrs-1.8.8.jar
- jackson-mapper-asl-1.8.8.jar
- jackson-module-scala_2.10-2.4.4.jar
- jackson-xc-1.8.8.jar
- jansi-1.4.jar
- javax.inject-1.jar
- javax.servlet-3.0.0.v201112011016.jar
- javax.servlet-3.1.jar
- javax.servlet-api-3.0.1.jar
- jaxb-api-2.2.2.jar
- jaxb-impl-2.2.3-1.jar
- jcl-over-slf4j-1.7.10.jar
- jersey-client-1.9.jar
- jersey-core-1.9.jar
- jersey-grizzly2-1.9.jar
- jersey-guice-1.9.jar
- jersey-json-1.9.jar
- jersey-server-1.9.jar
- jersey-test-framework-core-1.9.jar
- jersey-test-framework-grizzly2-1.9.jar
- jets3t-0.7.1.jar
- jettison-1.1.jar
- jetty-util-6.1.26.jar
- jline-0.9.94.jar
- jline-2.10.4.jar
- jodd-core-3.6.3.jar
- json4s-ast_2.10-3.2.10.jar
- json4s-core_2.10-3.2.10.jar
- json4s-jackson_2.10-3.2.10.jar
- jsr305-1.3.9.jar
- jtransforms-2.4.0.jar
- jul-to-slf4j-1.7.10.jar
- kryo-2.21.jar
- log4j-1.2.17.jar
- lz4-1.2.0.jar
- management-api-3.0.0-b012.jar
- mesos-0.21.0-shaded-protobuf.jar
- metrics-core-3.1.0.jar
- metrics-graphite-3.1.0.jar
- metrics-json-3.1.0.jar
- metrics-jvm-3.1.0.jar
- minlog-1.2.jar
- netty-3.8.0.Final.jar
- netty-all-4.0.23.Final.jar
- objenesis-1.2.jar
- opencsv-2.3.jar
- oro-2.0.8.jar
- paranamer-2.6.jar
- parquet-column-1.6.0rc3.jar
- parquet-common-1.6.0rc3.jar
- parquet-encoding-1.6.0rc3.jar
- parquet-format-2.2.0-rc1.jar
- parquet-generator-1.6.0rc3.jar
- parquet-hadoop-1.6.0rc3.jar
- parquet-jackson-1.6.0rc3.jar
- protobuf-java-2.4.1.jar
- protobuf-java-2.5.0-spark.jar
- py4j-0.8.2.1.jar
- pyrolite-2.0.1.jar
- quasiquotes_2.10-2.0.1.jar
- reflectasm-1.07-shaded.jar
- scala-compiler-2.10.4.jar
- scala-library-2.10.4.jar
- scala-reflect-2.10.4.jar
- scalap-2.10.4.jar
- scalatest_2.10-2.2.1.jar
- slf4j-api-1.7.10.jar
- slf4j-log4j12-1.7.10.jar
- snappy-java-1.1.1.6.jar
- spark-bagel_2.10-1.4.0-SNAPSHOT.jar
- spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
- spark-core_2.10-1.4.0-SNAPSHOT.jar
- spark-graphx_2.10-1.4.0-SNAPSHOT.jar
- spark-launcher_2.10-1.4.0-SNAPSHOT.jar
- spark-mllib_2.10-1.4.0-SNAPSHOT.jar
- spark-network-common_2.10-1.4.0-SNAPSHOT.jar
- spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
- spark-repl_2.10-1.4.0-SNAPSHOT.jar
- spark-sql_2.10-1.4.0-SNAPSHOT.jar
- spark-streaming_2.10-1.4.0-SNAPSHOT.jar
- spire-macros_2.10-0.7.4.jar
- spire_2.10-0.7.4.jar
- stax-api-1.0.1.jar
- stream-2.7.0.jar
- tachyon-0.5.0.jar
- tachyon-client-0.5.0.jar
- uncommons-maths-1.2.2a.jar
- unused-1.0.0.jar
- xmlenc-0.52.jar
- xz-1.0.jar
- zookeeper-3.4.5.jar

srowen · 2015-04-13T22:07:49Z

Looking strong. I think that's all the comments addressed now. Going once, going twice?

andrewor14 · 2015-04-13T23:04:37Z

docs/running-on-yarn.md

@@ -48,9 +48,9 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
 </tr>
 <tr>
  <td><code>spark.yarn.am.waitTime</code></td>
-  <td>100000</td>
+  <td>100s</td>


unfortunately this needs to be 100000ms for backward compatibility, because we stated in the docs that these configs accept raw numeric values assuming the units of the default value.

(either that or we leave out that line L51 in configuration.md)

Andrew - I'd prefer to leave out that line. I think it makes things simpler and encourages transitioning to the new paradigm.

andrewor14 · 2015-04-13T23:27:30Z

Ok, LGTM I'm merging this into master. It's great to see things on your todo list getting ticked off by others in the community. Thanks @ilganeli @srowen @vanzin!

SparkQA · 2015-04-14T00:40:33Z

Test build #30204 has finished for PR 5236 at commit 4526c81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

* Switched to using newly added getTimeAsSeconds method (see apache#5236) * Renamed minRememberDuration to minRememberDurationS to be compatible with the examples in the pull request above.

rxin · 2015-04-29T05:35:30Z

I'm late to the party, but just want to say -- this is super cool!

ilganeli · 2015-04-29T05:38:49Z

Hah. Thanks.

Sent with Good (www.good.com)

-----Original Message-----
From: Reynold Xin [[email protected]:[email protected]]
Sent: Wednesday, April 29, 2015 01:36 AM Eastern Standard Time
To: apache/spark
Cc: Ganelin, Ilya
Subject: Re: [spark] [SPARK-5931][CORE] Use consistent naming for time properties (#5236)

I'm late to the party, but just want to say -- this is super cool!

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/5236#issuecomment-97312212.

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Ilya Ganelin added 17 commits March 27, 2015 14:15

[SPARK-5931] Updated Utils and JavaUtils classes to add helper method…

59bf9e1

…s to handle time strings. Updated time strings in a few places to properly parse time

Updated usage of spark.core.connection.ack.wait.timeout

404f8c3

Updated usage of spark.akka.timeout

7db6d2a

Updated usage of spark.storage.blockManagerSlaveTimeout

4933fda

Updated spark.shuffle.io.retryWait

c9f5cad

updated spark.shuffle.sasl.timeout

21ef3dd

Updated usage of spark.cleaner.ttl

064ebd6

updated spark.akka.heartbeat.interval

7320c87

Updated spark.locality.wait

272c215

Updated spark.scheduler.maxRegisteredResourcesWaitingTime

3352d34

Updated spark.scheduler.revive.interval

3f1cfc8

Upated spark.speculation.interval

6d1518e

Updated spark.dynamicAllocation.executorIdleTimeout

2fcc91c

Updated spark.dynamicAllocation.schedulerBacklogTimeout

5181597

Updated spark.core.connection.auth.wait.timeout

c6a0095

Updated spark.streaming.blockInterval

cde9bff

Updated configuration doc with note on specifying time properties

42477aa

ilganeli changed the title ~~[SPARK-5931][CORE][WIP] Use consistent naming for time properties~~ [SPARK-5931][CORE] Use consistent naming for time properties Mar 27, 2015

Fixed misuse of time in streaming context test

9a29d8d

ilganeli added 2 commits March 27, 2015 22:27

Update Utils.scala

34f87c2

Fixed ordering to fix overlap of s with ms and us.

Update JavaUtils.java

8f741e1

Fixed overlap of (s) with (ms) and (us) in cascading if.

ilganeli closed this Mar 28, 2015

ilganeli reopened this Mar 28, 2015

andrewor14 reviewed Apr 13, 2015
View reviewed changes

Made the most recent round of changes

951ca2d

andrewor14 reviewed Apr 13, 2015
View reviewed changes

Doc updates

f5fafcd

Fixing style errors

de3bff9

andrewor14 reviewed Apr 13, 2015
View reviewed changes

Update configuration.md

4526c81

asfgit closed this in c4ab255 Apr 13, 2015

This was referenced Apr 15, 2015

[SPARK-6046] [core] Reorganize deprecated config support in SparkConf. #5514

Closed

[SPARK-5338][MESOS] Add cluster mode support for Mesos #5144

Closed

[SPARK-5931][CORE] Use consistent naming for time properties #5236

[SPARK-5931][CORE] Use consistent naming for time properties #5236

Uh oh!

Conversation

ilganeli commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

ilganeli commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

andrewor14 Apr 13, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 13, 2015

Uh oh!

andrewor14 Apr 13, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 13, 2015

Uh oh!

andrewor14 commented Apr 13, 2015

Uh oh!

SparkQA commented Apr 13, 2015

Uh oh!

srowen commented Apr 13, 2015

Uh oh!

andrewor14 Apr 13, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 Apr 13, 2015

Choose a reason for hiding this comment

Uh oh!

ilganeli Apr 13, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 13, 2015

Uh oh!

SparkQA commented Apr 14, 2015

Uh oh!

rxin commented Apr 29, 2015

Uh oh!

ilganeli commented Apr 29, 2015

Uh oh!

Uh oh!