[SPARK-4573] [SQL] Add SettableStructObjectInspector support in "wrap" function #3429

chenghao-intel · 2014-11-24T12:59:01Z

Hive UDAF may create an customized object constructed by SettableStructObjectInspector, this is critical when integrate Hive UDAF with the refactor-ed UDAF interface.

Performance issue in wrap/unwrap since more match cases added, will do it in another PR.

SparkQA · 2014-11-24T13:02:51Z

Test build #23782 has started for PR 3429 at commit 72e4332.

This patch merges cleanly.

SparkQA · 2014-11-24T13:03:48Z

Test build #23782 has finished for PR 3429 at commit 72e4332.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-24T13:03:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23782/
Test FAILed.

SparkQA · 2014-11-25T07:59:58Z

Test build #23826 has started for PR 3429 at commit 932940d.

This patch merges cleanly.

SparkQA · 2014-11-25T08:14:45Z

Test build #23829 has started for PR 3429 at commit f1b6749.

This patch merges cleanly.

SparkQA · 2014-11-25T08:37:26Z

Test build #23826 has finished for PR 3429 at commit 932940d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-25T08:37:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23826/
Test FAILed.

SparkQA · 2014-11-25T08:51:50Z

Test build #23829 has finished for PR 3429 at commit f1b6749.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-25T08:51:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23829/
Test FAILed.

SparkQA · 2014-11-25T15:40:35Z

Test build #23844 has started for PR 3429 at commit 3ed284c.

This patch merges cleanly.

SparkQA · 2014-11-25T16:14:05Z

Test build #23844 has finished for PR 3429 at commit 3ed284c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-25T16:14:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23844/
Test FAILed.

SparkQA · 2014-11-26T02:05:19Z

Test build #23859 has started for PR 3429 at commit 2977e9b.

This patch merges cleanly.

SparkQA · 2014-11-26T03:16:26Z

Test build #23859 has finished for PR 3429 at commit 2977e9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-26T03:16:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23859/
Test PASSed.

marmbrus · 2014-12-02T00:03:26Z

Thanks for working on this and adding a bunch of tests. All of this is getting pretty complicated, so I think it would be good if you could add some more explanation to the scala doc of wrap/unwrap that discusses what types object inspectors exist and which will be returned for what type of expression.

marmbrus · 2014-12-02T00:06:08Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala

+    checkValues(d, unwrap(wrap(null, toInspector(Literal(d, dt))), toInspector(Literal(d, dt))))
+  }
+
+  test("wrap / unwrap #6") {


Maybe instead of #1, #2, etc name them "Maps", "Arrays", etc

SparkQA · 2014-12-02T15:35:14Z

Test build #24046 has started for PR 3429 at commit f5a40e8.

This patch merges cleanly.

SparkQA · 2014-12-02T16:38:50Z

Test build #24046 has finished for PR 3429 at commit f5a40e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-02T16:38:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24046/
Test PASSed.

SparkQA · 2014-12-03T08:47:38Z

Test build #24084 has started for PR 3429 at commit 2b0561d.

This patch merges cleanly.

SparkQA · 2014-12-03T09:55:40Z

Test build #24084 has finished for PR 3429 at commit 2b0561d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-03T09:55:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24084/
Test PASSed.

chenghao-intel · 2014-12-10T01:39:22Z

@marmbrus I've updated the scala doc a little bit, I know that's quite complicated now, but we need to make thing right (to integrated with Hive UDFs seamlessly), since bugs like #2802 is depended on this.

I will make another PR to refactor / improve the performance described at https://github.com/apache/spark/pull/3429/files#diff-f88c3e731fcb17b1323b778807c35b38R167 once this PR merged.

marmbrus · 2014-12-17T22:42:26Z

/cc @liancheng can you help review this?

I did a quick pass and this seems reasonable. @chenghao-intel is this ready to merge if @liancheng approves?

liancheng · 2014-12-18T08:01:55Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala

+ *     Array[Byte]
+ *     java.sql.Date
+ *     java.sql.Timestamp
+ *  Complicated Types =>


"Complicated" => "Complex"

liancheng · 2014-12-18T08:24:27Z

In general this LGTM except for some minor styling comments, thanks!

SparkQA · 2014-12-18T14:17:27Z

Test build #24590 has started for PR 3429 at commit 9f0aff3.

This patch merges cleanly.

chenghao-intel · 2014-12-18T14:17:50Z

Thank you @liancheng , I've updated the code as feedback. @marmbrus I think this PR is ready to be merged once Jenkins agrees too.

SparkQA · 2014-12-18T15:28:46Z

Test build #24590 has finished for PR 3429 at commit 9f0aff3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-18T15:28:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24590/
Test PASSed.

marmbrus · 2014-12-19T04:22:16Z

Thanks! Merged to master.

JoshRosen · 2014-12-19T06:58:47Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala

+      case x: ByteObjectInspector if x.preferWritable() => x.get(data)
+      case x: HiveDecimalObjectInspector => HiveShim.toCatalystDecimal(x, data)
+      case x: BinaryObjectInspector if x.preferWritable() =>
+        x.getPrimitiveWritableObject(data).copyBytes()


It looks like this PR may have caused a build break for the hadoop1.0 profile:

https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/1245/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=centos/

[warn] Note: Recompile with -Xlint:unchecked for details. [info] Compiling 21 Scala sources and 1 Java source to /home/jenkins/workspace/Spark-Master-SBT/AMPLAB_JENKINS_BUILD_PROFILE/hadoop1.0/label/centos/sql/hive/target/scala-2.10/classes... [error] /home/jenkins/workspace/Spark-Master-SBT/AMPLAB_JENKINS_BUILD_PROFILE/hadoop1.0/label/centos/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:300: value copyBytes is not a member of org.apache.hadoop.io.BytesWritable [error] x.getPrimitiveWritableObject(data).copyBytes() [error] ^ [error] one error found [error] (hive/compile:compile) Compilation failed [error] Total time: 117 s, completed Dec 18, 2014 8:51:43 PM [error] Got a return code of 1 on line 155 of the run-tests script. Build step 'Execute shell' marked build as failure

Since this didn't break the pull request builder and it's nighttime now (so we're probably not merging tons of stuff), I'm going to hold off on reverting this for a little bit to see if we can come up with a quick hotfix. Otherwise, I'll revert this commit when I get up tomorrow.

Thank you @JoshRosen , I've created #3742 for this hot fix.

I've merged that PR, so the build should be fixed. Thanks!

Since #3429 has been merged, the bug of wrapping to Writable for HiveGenericUDF is resolved, we can safely remove the foldable checking in `HiveGenericUdf.eval`, which discussed in #2802. Author: Cheng Hao <[email protected]> Closes #3745 from chenghao-intel/generic_udf and squashes the following commits: 622ad03 [Cheng Hao] Remove the unnecessary code change in Generic UDF

Add settable StructObjectInspector support

72e4332

Add more unit test

932940d

Update the comment

f1b6749

chenghao-intel mentioned this pull request Nov 25, 2014

[SPARK-4233] [SQL] WIP:Simplify the UDAF API (Interface) #3247

Closed

3 tasks

fix the date type comparison

3ed284c

remove the timezone setting for test suite

2977e9b

marmbrus reviewed Dec 2, 2014
View reviewed changes

chenghao-intel mentioned this pull request Dec 2, 2014

[SPARK-3891][SQL] Add array support to percentile, percentile_approx and constant inspectors support #2802

Closed

add scala doc

f5a40e8

Add more scala doc

2b0561d

liancheng reviewed Dec 18, 2014
View reviewed changes

update code style issues as feedbacks

9f0aff3

asfgit closed this in ae9f128 Dec 19, 2014

JoshRosen reviewed Dec 19, 2014
View reviewed changes

chenghao-intel mentioned this pull request Dec 19, 2014

[SPARK-4904] [SQL] Remove the unnecessary code change in Generic UDF #3745

Closed

chenghao-intel deleted the settable_oi branch January 21, 2015 12:47

[SPARK-4573] [SQL] Add SettableStructObjectInspector support in "wrap" function #3429

[SPARK-4573] [SQL] Add SettableStructObjectInspector support in "wrap" function #3429

Uh oh!

Conversation

chenghao-intel commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

AmplabJenkins commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

AmplabJenkins commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

AmplabJenkins commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 25, 2014

Uh oh!

AmplabJenkins commented Nov 25, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

AmplabJenkins commented Nov 26, 2014

Uh oh!

marmbrus commented Dec 2, 2014

Uh oh!

marmbrus Dec 2, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 2, 2014

Uh oh!

SparkQA commented Dec 2, 2014

Uh oh!

AmplabJenkins commented Dec 2, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

AmplabJenkins commented Dec 3, 2014

Uh oh!

chenghao-intel commented Dec 10, 2014

Uh oh!

marmbrus commented Dec 17, 2014

Uh oh!

liancheng Dec 18, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

chenghao-intel commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

AmplabJenkins commented Dec 18, 2014

Uh oh!

marmbrus commented Dec 19, 2014

Uh oh!

JoshRosen Dec 19, 2014

Choose a reason for hiding this comment

Uh oh!

JoshRosen Dec 19, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel Dec 19, 2014

Choose a reason for hiding this comment