[SPARK-12700] [SQL] embed condition into SMJ and BroadcastHashJoin #10653
Conversation
|
Test build #48984 has finished for PR 10653 at commit
|
|
Test build #48986 has finished for PR 10653 at commit
|
| @@ -78,8 +78,11 @@ trait HashOuterJoin { | |||
|
|
|||
| @transient private[this] lazy val leftNullRow = new GenericInternalRow(left.output.length) | |||
| @transient private[this] lazy val rightNullRow = new GenericInternalRow(right.output.length) | |||
| @transient private[this] lazy val boundCondition = | |||
| @transient private[this] lazy val boundCondition = if (condition.isDefined) { | |||
There was a problem hiding this comment.
It's not related to this PR, just make it to aligned with others (a little faster for empty condition).
|
@nongli Do you have more comments on this one? |
|
LGTM. Let's make sure we don't need this when whole stage codegen is further along. |
|
For SMJ, we may not support whole stage codegen, this will be useful. For BroadcastHashJoin, once whole stage codegen is supported, we should not reply on this. Merging this into master, thanks. |
|
@davies I think it's reasonable to codegen the merge part of SMJ and have this start a new pipeline for whole stage codegen. Not required for the first go but shouldn't be too far away. |
|
@nongli That sounds good. |
Currently SortMergeJoin and BroadcastHashJoin do not support condition, the need a followed Filter for that, the result projection to generate UnsafeRow could be very expensive if they generate lots of rows and could be filtered mostly by condition.
This PR brings the support of condition for SortMergeJoin and BroadcastHashJoin, just like other outer joins do.
This could improve the performance of Q72 by 7x (from 120s to 16.5s).