Skip to content

Commit 1a6f6a4

Browse files
yifeihbulldozer-bot[bot]
authored andcommitted
[SPARK-27514][SQL] Skip collapsing windows with empty window expressions (apache#538)
## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain) github: apache#24411 jira: https://issues.apache.org/jira/browse/SPARK-27514 ## What changes were proposed in this pull request? A previous change moved the removal of empty window expressions to the RemoveNoopOperations rule, which comes after the CollapseWindow rule. Therefore, by the time we get to CollapseWindow, we aren't guaranteed that empty windows have been removed. This change checks that the window expressions are not empty, and only collapses the windows if both windows are non-empty. A lengthier description and repro steps here: https://issues.apache.org/jira/browse/SPARK-27514 ## How was this patch tested? A unit test, plus I reran the breaking case mentioned in the Jira ticket.
1 parent f2b22a1 commit 1a6f6a4

File tree

2 files changed

+12
-0
lines changed

2 files changed

+12
-0
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -798,6 +798,7 @@ object CollapseWindow extends Rule[LogicalPlan] {
798798
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
799799
case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
800800
if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
801+
we1.nonEmpty && we2.nonEmpty &&
801802
// This assumes Window contains the same type of window expressions. This is ensured
802803
// by ExtractWindowFunctions.
803804
WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseWindowSuite.scala

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,4 +89,15 @@ class CollapseWindowSuite extends PlanTest {
8989
val optimized = Optimize.execute(query.analyze)
9090
comparePlans(optimized, expected)
9191
}
92+
93+
test("Skip windows with empty window expressions") {
94+
val query = testRelation
95+
.window(Seq(), partitionSpec1, orderSpec1)
96+
.window(Seq(sum(a).as('sum_a)), partitionSpec1, orderSpec1)
97+
98+
val optimized = Optimize.execute(query.analyze)
99+
val correctAnswer = query.analyze
100+
101+
comparePlans(optimized, correctAnswer)
102+
}
92103
}

0 commit comments

Comments
 (0)