Skip to content

[SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75/q77 #19969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75/q77 #19969

wants to merge 1 commit into from

Conversation

maropu
Copy link
Member

@maropu maropu commented Dec 13, 2017 •
edited
Loading

What changes were proposed in this pull request?

This pr fixed a compilation error of TPCDS q75/q77 caused by #19813;

  java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 371, Column 16: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 371, Column 16: Expression "bhj_matched" is not an rvalue
  at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
  at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)
  at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
  at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)

How was this patch tested?

Manually checked q75/q77 can be properly compiled

@maropu
Copy link
Member Author

maropu commented Dec 13, 2017

cc: @viirya @cloud-fan @kiszk

@maropu
Copy link
Member Author

maropu commented Dec 13, 2017

Also, I feel we'd be better to check if all the tpcds queries can be properly compiled in tests like;
master...maropu:SPARK-22140-FOLLOWUP
WDYT, guys?

@cloud-fan
Copy link
Contributor

hmm, didn't we already check it in tests? WholeStageExec will report compile error in test environment.

cloud-fan
cloud-fan reviewed Dec 13, 2017
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
@@ -192,7 +192,7 @@ case class BroadcastHashJoinExec(
| $value = ${ev.value};
|}
""".stripMargin
ExprCode(code, isNull, value)
ExprCode(code, isNull, value, inputRow = matched)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to check more places that create ExprCode outside of Expression.genCode and see if we set the inputs correctly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya, I'll look around.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked if the files below have the same issue though, I didn't found it (double-checks needed);

./catalyst/expressions/codegen/CodeGenerator.scala:202:    ExprCode(code, "false", value)
./catalyst/expressions/codegen/ExpressionCodegen.scala:89:      ExprCode(code = "", value = state.value, isNull = state.isNull)
./catalyst/expressions/codegen/GenerateMutableProjection.scala:90:        val ev = ExprCode("", s"isNull_$i", s"value_$i")
./catalyst/expressions/codegen/GenerateSafeProjection.scala:77:    ExprCode(code, "false", output)
./catalyst/expressions/codegen/GenerateSafeProjection.scala:107:    ExprCode(code, "false", output)
./catalyst/expressions/codegen/GenerateSafeProjection.scala:128:    ExprCode(code, "false", output)
./catalyst/expressions/codegen/GenerateSafeProjection.scala:140:    case _ => ExprCode("", "false", input)
./catalyst/expressions/codegen/GenerateUnsafeProjection.scala:55:      ExprCode("", s"$tmpInput.isNullAt($i)", ctx.getValue(tmpInput, dt, i.toString))
./catalyst/expressions/codegen/GenerateUnsafeProjection.scala:353:    ExprCode(code, "false", result)
./catalyst/expressions/complexTypeCreator.scala:397:    ExprCode(code = eval.code, isNull = "false", value = eval.value)
./catalyst/expressions/datetimeExpressions.scala:677:          ExprCode("", "true", ctx.defaultValue(dataType))
./catalyst/expressions/datetimeExpressions.scala:812:        ExprCode("", "true", "(UTF8String) null")
./catalyst/expressions/Expression.scala:103:      ExprCode(ctx.registerComment(this.toString), subExprState.isNull, subExprState.value)
./catalyst/expressions/Expression.scala:107:      val eval = doGenCode(ctx, ExprCode("", isNull, value))
./catalyst/expressions/misc.scala:85:    ExprCode(code = s"""${eval.code}
./catalyst/expressions/nullExpressions.scala:325:    ExprCode(code = eval.code, isNull = "false", value = eval.isNull)
./catalyst/expressions/nullExpressions.scala:351:    ExprCode(code = eval.code, isNull = "false", value = s"(!(${eval.isNull}))")
./catalyst/expressions/objects/objects.scala:448:    ExprCode(code = "", value = value, isNull = if (nullable) isNull else "false")
./execution/aggregate/HashAggregateExec.scala:200:      ExprCode(ev.code + initVars, isNull, value)
./execution/aggregate/HashMapGenerator.scala:59:      ExprCode(ev.code + initVars, isNull, value)
./execution/basicPhysicalOperators.scala:372:    val ev = ExprCode("", "false", value)
./execution/ColumnarBatchScan.scala:62:    ExprCode(code, isNullVar, valueVar)
./execution/ExpandExec.scala:159:        ExprCode(code, isNull, value)
./execution/GenerateExec.scala:172:        Seq(ExprCode("", s"$index == -1", index))
./execution/GenerateExec.scala:174:        Seq(ExprCode("", "false", index))
./execution/GenerateExec.scala:317:      ExprCode(code, isNull, value)
./execution/GenerateExec.scala:319:      ExprCode(s"$javaType $value = $getter;", "false", value)
./execution/joins/BroadcastHashJoinExec.scala:195:        ExprCode(code, isNull, value, inputRow = matched)
./execution/joins/BroadcastHashJoinExec.scala:490:    val resultVar = input ++ Seq(ExprCode("", "false", existsVar))
./execution/joins/SortMergeJoinExec.scala:535:        (ExprCode(code, isNull, value), leftVarsDecl)
./execution/joins/SortMergeJoinExec.scala:539:        (ExprCode(code, "false", value), leftVarsDecl)
./execution/WholeStageCodegenExec.scala:130:      ExprCode("", "false", row)
./execution/WholeStageCodegenExec.scala:145:        ExprCode(code, "false", ev.value)
./execution/WholeStageCodegenExec.scala:148:        ExprCode("", "false", "unsafeRow")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except for L195, BroadcastHashJoinExec is safe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've double-checked the above list. I think they are safe.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@maropu
Copy link
Member Author

maropu commented Dec 13, 2017 •
edited
Loading

TPCDSQuerySuite actually doesn't compile and run these queries in master, so it didn't report that.

@maropu
Copy link
Member Author

maropu commented Dec 13, 2017 •
edited
Loading

If no problem, I'll include the changes for additional tests in this pr.

@SparkQA
Copy link

SparkQA commented Dec 13, 2017

Test build #84870 has finished for PR 19969 at commit 777d585.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Member

How about q77?

@gatorsmile
Copy link
Member

See the result of #19971

@maropu
Copy link
Member Author

maropu commented Dec 13, 2017

ya, this pr also fixes q77.

@maropu maropu changed the title [SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75 [SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75/q77 Dec 13, 2017
@viirya
Copy link
Member

viirya commented Dec 13, 2017

@maropu Thanks for finding and fixing this. I'll look into the possible places later (just gets up).

@gatorsmile
Copy link
Member

gatorsmile commented Dec 13, 2017 •
edited
Loading

I am still worrying about the original PR #19813. We also need a conf to turn that off.

@gatorsmile
Copy link
Member

LGTM merged to master.

@asfgit asfgit closed this in ef92999 Dec 13, 2017
@viirya viirya mentioned this pull request Dec 14, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@viirya viirya viirya left review comments

@cloud-fan cloud-fan cloud-fan left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@maropu @cloud-fan @SparkQA @gatorsmile @viirya