[SPARK-7662] [SQL] Resolve correct names for generator in projection #6178

chenghao-intel · 2015-05-15T07:17:15Z

select explode(map(value, key)) from src;

Throws exception

org.apache.spark.sql.AnalysisException: The number of aliases supplied in the AS clause does not match the number of columns output by the UDTF expected 2 aliases but got _c0 ;
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38)
at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:43)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveGenerate$$makeGeneratorOutput(Analyzer.scala:605)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:562)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:548)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:548)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:538)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:222)

AmplabJenkins · 2015-05-15T07:22:10Z

Merged build triggered.

AmplabJenkins · 2015-05-15T07:22:15Z

Merged build started.

SparkQA · 2015-05-15T07:23:10Z

Test build #32795 has started for PR 6178 at commit 19db09e.

SparkQA · 2015-05-15T08:27:35Z

Test build #32795 has finished for PR 6178 at commit 19db09e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-15T08:27:39Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-15T08:27:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32795/
Test FAILed.

chenghao-intel · 2015-05-15T16:06:34Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

    }

    intercept[AnalysisException] {
-      sql("SELECT explode(map(key, value)) as k1, k2, key FROM src").collect()
+      sql("SELECT explode(array(key, key)) as k1, explode(array(key, key)) FROM src").collect()


It's an illegal SQL for

SELECT explode(map(key, value)) as k1, k2, key FROM src

But

SELECT explode(map(key, value)) as (k1, k2), key FROM src

AmplabJenkins · 2015-05-15T16:07:11Z

Merged build triggered.

AmplabJenkins · 2015-05-15T16:07:18Z

Merged build started.

SparkQA · 2015-05-15T16:09:06Z

Test build #32834 has started for PR 6178 at commit e1d93ab.

SparkQA · 2015-05-15T17:08:12Z

Test build #32834 has finished for PR 6178 at commit e1d93ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-15T17:08:16Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-15T17:08:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32834/
Test FAILed.

marmbrus · 2015-05-15T19:28:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

@@ -576,6 +576,13 @@ class Analyzer(
    /** Extracts a [[Generator]] expression and any names assigned by aliases to their output. */
    private object AliasedGenerator {
      def unapply(e: Expression): Option[(Generator, Seq[String])] = e match {
+        case Alias(g: Generator, name) if g.elementTypes.size > 1 => {


Seems we also need to check that the name is the default.

Yes, agree, we should throw exception if it's not default name here. But seems very difficult to tell whether it's default name or not, as in HiveQl, it will named each of the column anyway for Project.
Any simple way for this?

I think the real fix is to stop setting defaults in in the parser and instead let the analyzer do it (as we do with MultiAlias today). However, thats a pretty big change, so I'd be okay just checking with a regex here for now.

Sorry, I am not sure what you mean for regex? You mean to check if the name like _c0, _c1 in regex?

Yeah, that's what I was thinking.
On May 18, 2015 6:00 PM, "Cheng Hao" [email protected] wrote:

In
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
#6178 (comment):

@@ -576,6 +576,13 @@ class Analyzer(
/** Extracts a [[Generator]] expression and any names assigned by aliases to their output. */
private object AliasedGenerator {
def unapply(e: Expression): Option[(Generator, Seq[String])] = e match {

case Alias(g: Generator, name) if g.elementTypes.size > 1 => {

Sorry, I am not sure what you mean for regex? You mean to check if the
name like _c0, _c1 in regex?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/6178/files#r30561456.

AmplabJenkins · 2015-05-18T06:57:10Z

Merged build triggered.

AmplabJenkins · 2015-05-18T06:57:20Z

Merged build started.

SparkQA · 2015-05-18T06:59:11Z

Test build #32975 has started for PR 6178 at commit 5c3f2c5.

SparkQA · 2015-05-18T07:09:26Z

Test build #32975 has finished for PR 6178 at commit 5c3f2c5.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T07:09:29Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-18T07:09:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32975/
Test FAILed.

chenghao-intel · 2015-05-18T07:20:59Z

Weird? Failed in MiMa Test.
retest this please.

chenghao-intel · 2015-05-18T08:30:22Z

retest this please

AmplabJenkins · 2015-05-18T08:32:14Z

Merged build triggered.

AmplabJenkins · 2015-05-18T08:32:20Z

Merged build started.

SparkQA · 2015-05-18T08:33:16Z

Test build #32983 has started for PR 6178 at commit 5c3f2c5.

SparkQA · 2015-05-18T10:26:52Z

Test build #32983 has finished for PR 6178 at commit 5c3f2c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T10:26:57Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-18T10:26:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32983/
Test PASSed.

marmbrus · 2015-05-18T19:24:09Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

-  test("multiple generator in projection") {
+  test("multiple generators in projection") {
+    // TGF with non-TGF in project is allowed in Spark SQL, but not in Hive
+    sql("SELECT explode(map(key, key)) as (k1, k2), key, value FROM src").collect()


If we don't have a test for this elsewhere, then this should probably be in its own test block and use checkAnswer.

AmplabJenkins · 2015-05-19T02:37:10Z

Merged build triggered.

AmplabJenkins · 2015-05-19T02:37:19Z

Merged build started.

SparkQA · 2015-05-19T02:39:19Z

Test build #33046 has started for PR 6178 at commit 916fbe9.

SparkQA · 2015-05-19T04:32:48Z

Test build #33046 has finished for PR 6178 at commit 916fbe9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-19T04:32:52Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-19T04:32:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33046/
Test PASSed.

``` select explode(map(value, key)) from src; ``` Throws exception ``` org.apache.spark.sql.AnalysisException: The number of aliases supplied in the AS clause does not match the number of columns output by the UDTF expected 2 aliases but got _c0 ; at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38) at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:43) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveGenerate$$makeGeneratorOutput(Analyzer.scala:605) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:562) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:548) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:548) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:538) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:222) ``` Author: Cheng Hao <[email protected]> Closes #6178 from chenghao-intel/explode and squashes the following commits: 916fbe9 [Cheng Hao] add more strict rules for TGF alias 5c3f2c5 [Cheng Hao] fix bug in unit test e1d93ab [Cheng Hao] Add more unit test 19db09e [Cheng Hao] resolve names for generator in projection (cherry picked from commit bcb1ff8) Signed-off-by: Michael Armbrust <[email protected]>

marmbrus · 2015-05-19T22:27:12Z

Thanks! Merged to master and branch-1.4.

``` select explode(map(value, key)) from src; ``` Throws exception ``` org.apache.spark.sql.AnalysisException: The number of aliases supplied in the AS clause does not match the number of columns output by the UDTF expected 2 aliases but got _c0 ; at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38) at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:43) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveGenerate$$makeGeneratorOutput(Analyzer.scala:605) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:562) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16$$anonfun$22.apply(Analyzer.scala:548) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:548) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate$$anonfun$apply$16.applyOrElse(Analyzer.scala:538) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:222) ``` Author: Cheng Hao <[email protected]> Closes apache#6178 from chenghao-intel/explode and squashes the following commits: 916fbe9 [Cheng Hao] add more strict rules for TGF alias 5c3f2c5 [Cheng Hao] fix bug in unit test e1d93ab [Cheng Hao] Add more unit test 19db09e [Cheng Hao] resolve names for generator in projection

resolve names for generator in projection

19db09e

Add more unit test

e1d93ab

chenghao-intel reviewed May 15, 2015
View reviewed changes

marmbrus reviewed May 15, 2015
View reviewed changes

fix bug in unit test

5c3f2c5

marmbrus reviewed May 18, 2015
View reviewed changes

add more strict rules for TGF alias

916fbe9

asfgit closed this in bcb1ff8 May 19, 2015

chenghao-intel deleted the explode branch July 2, 2015 08:31

[SPARK-7662] [SQL] Resolve correct names for generator in projection #6178

[SPARK-7662] [SQL] Resolve correct names for generator in projection #6178

Uh oh!

Conversation

chenghao-intel commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

SparkQA commented May 15, 2015

Uh oh!

SparkQA commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

chenghao-intel May 15, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

SparkQA commented May 15, 2015

Uh oh!

SparkQA commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

AmplabJenkins commented May 15, 2015

Uh oh!

marmbrus May 15, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intel May 18, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus May 18, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intel May 19, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus May 19, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

chenghao-intel commented May 18, 2015

Uh oh!

chenghao-intel commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

marmbrus May 18, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 19, 2015