Skip to content

Commit c1f3c27

Browse files
scwfrxin
authored andcommitted
[SPARK-4937][SQL] Comment for the newly optimization rules in BooleanSimplification
Follow up of #3778 /cc rxin Author: scwf <[email protected]> Closes #4086 from scwf/commentforspark-4937 and squashes the following commits: aaf89f6 [scwf] code style issue 2d3406e [scwf] added comment for spark-4937
1 parent f3bfc76 commit c1f3c27

File tree

1 file changed

+16
-2
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer

1 file changed

+16
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -311,13 +311,20 @@ object BooleanSimplification extends Rule[LogicalPlan] with PredicateHelper {
311311
// a && a => a
312312
case (l, r) if l fastEquals r => l
313313
case (_, _) =>
314+
/* Do optimize for predicates using formula (a || b) && (a || c) => a || (b && c)
315+
* 1. Split left and right to get the disjunctive predicates,
316+
* i.e. lhsSet = (a, b), rhsSet = (a, c)
317+
* 2. Find the common predict between lhsSet and rhsSet, i.e. common = (a)
318+
* 3. Remove common predict from lhsSet and rhsSet, i.e. ldiff = (b), rdiff = (c)
319+
* 4. Apply the formula, get the optimized predict: common || (ldiff && rdiff)
320+
*/
314321
val lhsSet = splitDisjunctivePredicates(left).toSet
315322
val rhsSet = splitDisjunctivePredicates(right).toSet
316323
val common = lhsSet.intersect(rhsSet)
317324
val ldiff = lhsSet.diff(common)
318325
val rdiff = rhsSet.diff(common)
319326
if (ldiff.size == 0 || rdiff.size == 0) {
320-
// a && (a || b)
327+
// a && (a || b) => a
321328
common.reduce(Or)
322329
} else {
323330
// (a || b || c || ...) && (a || b || d || ...) && (a || b || e || ...) ... =>
@@ -339,13 +346,20 @@ object BooleanSimplification extends Rule[LogicalPlan] with PredicateHelper {
339346
// a || a => a
340347
case (l, r) if l fastEquals r => l
341348
case (_, _) =>
349+
/* Do optimize for predicates using formula (a && b) || (a && c) => a && (b || c)
350+
* 1. Split left and right to get the conjunctive predicates,
351+
* i.e. lhsSet = (a, b), rhsSet = (a, c)
352+
* 2. Find the common predict between lhsSet and rhsSet, i.e. common = (a)
353+
* 3. Remove common predict from lhsSet and rhsSet, i.e. ldiff = (b), rdiff = (c)
354+
* 4. Apply the formula, get the optimized predict: common && (ldiff || rdiff)
355+
*/
342356
val lhsSet = splitConjunctivePredicates(left).toSet
343357
val rhsSet = splitConjunctivePredicates(right).toSet
344358
val common = lhsSet.intersect(rhsSet)
345359
val ldiff = lhsSet.diff(common)
346360
val rdiff = rhsSet.diff(common)
347361
if ( ldiff.size == 0 || rdiff.size == 0) {
348-
// a || (b && a)
362+
// a || (b && a) => a
349363
common.reduce(And)
350364
} else {
351365
// (a && b && c && ...) || (a && b && d && ...) || (a && b && e && ...) ... =>

0 commit comments

Comments
 (0)