Skip to content

Commit ba84329

Browse files
wangyumgatorsmile
authored andcommitted
[SPARK-21790][TESTS][FOLLOW-UP] Add filter pushdown verification back.
## What changes were proposed in this pull request? The previous PR(#19000) removed filter pushdown verification, This PR add them back. ## How was this patch tested? manual tests Author: Yuming Wang <[email protected]> Closes #19002 from wangyum/SPARK-21790-follow-up.
1 parent 988b84d commit ba84329

File tree

1 file changed

+13
-0
lines changed

1 file changed

+13
-0
lines changed

external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import java.util.Properties
2222
import java.math.BigDecimal
2323

2424
import org.apache.spark.sql.Row
25+
import org.apache.spark.sql.execution.{WholeStageCodegenExec, RowDataSourceScanExec}
2526
import org.apache.spark.sql.test.SharedSQLContext
2627
import org.apache.spark.sql.types._
2728
import org.apache.spark.tags.DockerTest
@@ -255,6 +256,18 @@ class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSQLCo
255256
val df = dfRead.filter(dfRead.col("date_type").lt(dt))
256257
.filter(dfRead.col("timestamp_type").lt(ts))
257258

259+
val parentPlan = df.queryExecution.executedPlan
260+
assert(parentPlan.isInstanceOf[WholeStageCodegenExec])
261+
val node = parentPlan.asInstanceOf[WholeStageCodegenExec]
262+
val metadata = node.child.asInstanceOf[RowDataSourceScanExec].metadata
263+
// The "PushedFilters" part should exist in Dataframe's
264+
// physical plan and the existence of right literals in
265+
// "PushedFilters" is used to prove that the predicates
266+
// pushing down have been effective.
267+
assert(metadata.get("PushedFilters").isDefined)
268+
assert(metadata("PushedFilters").contains(dt.toString))
269+
assert(metadata("PushedFilters").contains(ts.toString))
270+
258271
val row = df.collect()(0)
259272
assert(row.getDate(0).equals(dateVal))
260273
assert(row.getTimestamp(1).equals(timestampVal))

0 commit comments

Comments
 (0)