[SPARK-3412][SQL]add missing row api #2529

adrian-wang · 2014-09-25T02:14:53Z

@chenghao-intel assigned this to me, check PR #2284 for previous discussion

SparkQA · 2014-09-25T02:19:26Z

QA tests have started for PR 2529 at commit 1614493.

This patch merges cleanly.

SparkQA · 2014-09-25T03:09:04Z

QA tests have finished for PR 2529 at commit 1614493.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-25T03:09:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20776/

marmbrus · 2014-09-26T18:59:44Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Row.scala

@@ -98,6 +99,7 @@ trait MutableRow extends Row {
  def setByte(ordinal: Int, value: Byte)
  def setFloat(ordinal: Int, value: Float)
  def setString(ordinal: Int, value: String)
+  def setAs[T](ordinal: Int, value: T) = update(ordinal, value)


I don't think there is any reason to have this function. The update call works independent of type already.

marmbrus · 2014-09-26T19:03:24Z

Thanks for working on this! A few minor comments.

SparkQA · 2014-09-28T09:04:35Z

QA tests have started for PR 2529 at commit 4c18c29.

This patch merges cleanly.

SparkQA · 2014-09-28T11:04:36Z

Tests timed out after a configured wait of 120m.

AmplabJenkins · 2014-09-28T11:04:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20934/

adrian-wang · 2014-09-30T05:22:47Z

retest this please.

AmplabJenkins · 2014-09-30T05:37:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21021/

SparkQA · 2014-09-30T05:44:08Z

QA tests have started for PR 2529 at commit 4c18c29.

This patch merges cleanly.

adrian-wang · 2014-09-30T05:54:25Z

retest this please.

AmplabJenkins · 2014-09-30T06:07:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21023/

SparkQA · 2014-09-30T06:14:05Z

QA tests have started for PR 2529 at commit 4c18c29.

This patch merges cleanly.

SparkQA · 2014-09-30T06:36:22Z

QA tests have finished for PR 2529 at commit 4c18c29.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-30T07:04:41Z

QA tests have finished for PR 2529 at commit 4c18c29.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-10-01T22:00:18Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificRow.scala

@@ -306,4 +306,8 @@ final class SpecificMutableRow(val values: Array[MutableValue]) extends MutableR
  override def getByte(i: Int): Byte = {
    values(i).asInstanceOf[MutableByte].value
  }
+
+  override def getAs[T](i: Int): T = {
+    values(i).asInstanceOf[MutableAny].value.asInstanceOf[T]


You don't want the first .asInstanceOfhere as it can cause errors.

scala> new SpecificMutableRow(IntegerType :: Nil).getAs[Int](0) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableInt cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableAny

Since all MutableValue classes have a .value method I think you can just leave it out.

Also, looks like this file has always been named wrong. Would you mind fixing that (to SpecificMutableRow.scala) while you are at it?

marmbrus · 2014-10-01T22:01:37Z

Only one small change then this LGTM.

SparkQA · 2014-10-08T06:24:35Z

QA tests have started for PR 2529 at commit 7b7e6e3.

This patch merges cleanly.

AmplabJenkins · 2014-10-08T06:27:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21448/Test FAILed.

adrian-wang · 2014-10-08T06:38:07Z

retest this please.

SparkQA · 2014-10-08T06:44:47Z

QA tests have started for PR 2529 at commit 7b7e6e3.

This patch merges cleanly.

SparkQA · 2014-10-08T07:09:14Z

QA tests have finished for PR 2529 at commit 7b7e6e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-08T07:09:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21449/Test PASSed.

SparkQA · 2014-10-08T07:17:14Z

QA tests have finished for PR 2529 at commit 7b7e6e3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-08T07:17:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21451/Test FAILed.

adrian-wang · 2014-10-08T07:31:04Z

retest this please.

SparkQA · 2014-10-08T07:34:38Z

QA tests have started for PR 2529 at commit 7b7e6e3.

This patch merges cleanly.

SparkQA · 2014-10-08T08:20:46Z

QA tests have finished for PR 2529 at commit 7b7e6e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-08T08:20:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21453/Test PASSed.

marmbrus · 2014-10-09T00:25:55Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificMutableRow.scala

@@ -306,4 +306,15 @@ final class SpecificMutableRow(val values: Array[MutableValue]) extends MutableR
  override def getByte(i: Int): Byte = {
    values(i).asInstanceOf[MutableByte].value
  }
+
+  override def getAs[T](i: Int): T = values(i) match {


Can't this just be override def getAs[T](i: Int): T = values(i).boxed.asInstanceOf[T]?

And I'm sorry for going back again. I should have said boxed instead of value in my previous comment.

Sorry for miss that, I didn't notice .boxed method there...

SparkQA · 2014-10-09T02:00:12Z

QA tests have started for PR 2529 at commit c6594b2.

This patch merges cleanly.

SparkQA · 2014-10-09T02:44:29Z

QA tests have finished for PR 2529 at commit c6594b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-09T02:44:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21510/Test PASSed.

marmbrus · 2014-10-09T22:01:47Z

Thanks! Merged to master.

add missing row api

1614493

marmbrus reviewed Sep 26, 2014
View reviewed changes

remove setAs[T] and null judge

4c18c29

marmbrus reviewed Oct 1, 2014
View reviewed changes

adrian-wang added 2 commits October 7, 2014 23:04

rename file and refresh getAs[T]

7a39456

update pattern match

7b7e6e3

marmbrus reviewed Oct 9, 2014
View reviewed changes

using boxed

c6594b2

asfgit closed this in 0c0e09f Oct 9, 2014

[SPARK-3412][SQL]add missing row api #2529

[SPARK-3412][SQL]add missing row api #2529

Uh oh!

Conversation

adrian-wang commented Sep 25, 2014

Uh oh!

SparkQA commented Sep 25, 2014

Uh oh!

SparkQA commented Sep 25, 2014

Uh oh!

AmplabJenkins commented Sep 25, 2014

Uh oh!

marmbrus Sep 26, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Sep 26, 2014

Uh oh!

SparkQA commented Sep 28, 2014

Uh oh!

SparkQA commented Sep 28, 2014

Uh oh!

AmplabJenkins commented Sep 28, 2014

Uh oh!

adrian-wang commented Sep 30, 2014

Uh oh!

AmplabJenkins commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

adrian-wang commented Sep 30, 2014

Uh oh!

AmplabJenkins commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

SparkQA commented Sep 30, 2014

Uh oh!

marmbrus Oct 1, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Oct 1, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Oct 1, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

adrian-wang commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

adrian-wang commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

marmbrus Oct 9, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Oct 9, 2014

Choose a reason for hiding this comment

Uh oh!

adrian-wang Oct 9, 2014

Choose a reason for hiding this comment

Uh oh!