Skip to content

Commit 2c0f5de

Browse files
committed
[SC-4372] Set correct taskcontext in async-io thread
## What changes were proposed in this pull request? This patch fixes a broken test in `ParquetFilterSuite`. ## How was this patch tested? `ParquetFilterSuite` Author: Sameer Agarwal <[email protected]> Closes apache#68 from sameeragarwal/async-io.
1 parent 5c0ca48 commit 2c0f5de

File tree

2 files changed

+3
-1
lines changed

2 files changed

+3
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,10 @@ class FileScanRDD(
178178
}
179179

180180
def prepareNextFile(): Future[NextFile] = {
181+
val taskContext = TaskContext.get()
181182
if (files.hasNext) {
182183
Future {
184+
TaskContext.setTaskContext(taskContext)
183185
val nextFile = files.next()
184186
val nextFileIter =
185187
try {

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -563,7 +563,7 @@ class ParquetFilterSuite extends QueryTest with ParquetTest with SharedSQLContex
563563
}
564564
}
565565

566-
test("Fiters should be pushed down for vectorized Parquet reader at row group level") {
566+
test("Filters should be pushed down for vectorized Parquet reader at row group level") {
567567
import testImplicits._
568568

569569
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true",

0 commit comments

Comments
 (0)