Skip to content

Commit 7cc0f0e

Browse files
HyukjinKwondongjoon-hyun
authored andcommitted
[SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
### What changes were proposed in this pull request? See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109834/testReport/junit/org.apache.spark.sql/SQLQueryTestSuite/ ![Screen Shot 2019-08-28 at 4 08 58 PM](https://user-images.githubusercontent.com/6477701/63833484-2a23ea00-c9ae-11e9-91a1-0859cb183fea.png) ```xml <?xml version="1.0" encoding="UTF-8"?> <testsuite hostname="C02Y52ZLJGH5" name="org.apache.spark.sql.SQLQueryTestSuite" tests="3" errors="0" failures="0" skipped="0" time="14.475"> ... <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Scala UDF" time="6.703"> </testcase> <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Regular Python UDF" time="4.442"> </testcase> <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Scalar Pandas UDF" time="3.33"> </testcase> <system-out/> <system-err/> </testsuite> ``` Root cause seems a bug in SBT - it truncates the test name based on the last dot. sbt/sbt#2949 https://github.com/sbt/sbt/blob/v0.13.18/testing/src/main/scala/sbt/JUnitXmlTestsListener.scala#L71-L79 I tried to find a better way but couldn't find. Therefore, this PR proposes a workaround by appending the test file name into the assert log: ```diff [info] - inner-join.sql *** FAILED *** (4 seconds, 306 milliseconds) + [info] inner-join.sql [info] Expected "1 a [info] 1 a [info] 1 b [info] 1[]", but got "1 a [info] 1 a [info] 1 b [info] 1[ b]" Result did not match for query #6 [info] SELECT tb.* FROM ta INNER JOIN tb ON ta.a = tb.a AND ta.tag = tb.tag (SQLQueryTestSuite.scala:377) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528) ``` It will at least prevent us to search full logs to identify which test file is failed by clicking filed test. Note that this PR does not fully fix the issue but only fix the logs on its failed tests. ### Why are the changes needed? To debug Jenkins logs easier. Otherwise, we should open full logs and search which test was failed. ### Does this PR introduce any user-facing change? It will print out the file name of failed tests in Jenkins' test reports. ### How was this patch tested? Manually tested but Jenkins tests are required in this PR. Now it at least shows which file it is: ![Screen Shot 2019-08-30 at 10 16 32 PM](https://user-images.githubusercontent.com/6477701/64023705-de22a200-cb73-11e9-8806-2e98ad35adef.png) Closes #25630 from HyukjinKwon/SPARK-28894-1. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 3b07a4e commit 7cc0f0e

File tree

1 file changed

+35
-30
lines changed

1 file changed

+35
-30
lines changed

sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala

Lines changed: 35 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -342,39 +342,44 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
342342
stringToFile(resultFile, goldenOutput)
343343
}
344344

345-
// Read back the golden file.
346-
val expectedOutputs: Seq[QueryOutput] = {
347-
val goldenOutput = fileToString(new File(testCase.resultFile))
348-
val segments = goldenOutput.split("-- !query.+\n")
349-
350-
// each query has 3 segments, plus the header
351-
assert(segments.size == outputs.size * 3 + 1,
352-
s"Expected ${outputs.size * 3 + 1} blocks in result file but got ${segments.size}. " +
353-
s"Try regenerate the result files.")
354-
Seq.tabulate(outputs.size) { i =>
355-
QueryOutput(
356-
sql = segments(i * 3 + 1).trim,
357-
schema = segments(i * 3 + 2).trim,
358-
output = segments(i * 3 + 3).replaceAll("\\s+$", "")
359-
)
345+
// This is a temporary workaround for SPARK-28894. The test names are truncated after
346+
// the last dot due to a bug in SBT. This makes easier to debug via Jenkins test result
347+
// report. See SPARK-28894.
348+
withClue(s"${testCase.name}${System.lineSeparator()}") {
349+
// Read back the golden file.
350+
val expectedOutputs: Seq[QueryOutput] = {
351+
val goldenOutput = fileToString(new File(testCase.resultFile))
352+
val segments = goldenOutput.split("-- !query.+\n")
353+
354+
// each query has 3 segments, plus the header
355+
assert(segments.size == outputs.size * 3 + 1,
356+
s"Expected ${outputs.size * 3 + 1} blocks in result file but got ${segments.size}. " +
357+
s"Try regenerate the result files.")
358+
Seq.tabulate(outputs.size) { i =>
359+
QueryOutput(
360+
sql = segments(i * 3 + 1).trim,
361+
schema = segments(i * 3 + 2).trim,
362+
output = segments(i * 3 + 3).replaceAll("\\s+$", "")
363+
)
364+
}
360365
}
361-
}
362366

363-
// Compare results.
364-
assertResult(expectedOutputs.size, s"Number of queries should be ${expectedOutputs.size}") {
365-
outputs.size
366-
}
367-
368-
outputs.zip(expectedOutputs).zipWithIndex.foreach { case ((output, expected), i) =>
369-
assertResult(expected.sql, s"SQL query did not match for query #$i\n${expected.sql}") {
370-
output.sql
371-
}
372-
assertResult(expected.schema,
373-
s"Schema did not match for query #$i\n${expected.sql}: $output") {
374-
output.schema
367+
// Compare results.
368+
assertResult(expectedOutputs.size, s"Number of queries should be ${expectedOutputs.size}") {
369+
outputs.size
375370
}
376-
assertResult(expected.output, s"Result did not match for query #$i\n${expected.sql}") {
377-
output.output
371+
372+
outputs.zip(expectedOutputs).zipWithIndex.foreach { case ((output, expected), i) =>
373+
assertResult(expected.sql, s"SQL query did not match for query #$i\n${expected.sql}") {
374+
output.sql
375+
}
376+
assertResult(expected.schema,
377+
s"Schema did not match for query #$i\n${expected.sql}: $output") {
378+
output.schema
379+
}
380+
assertResult(expected.output, s"Result did not match for query #$i\n${expected.sql}") {
381+
output.output
382+
}
378383
}
379384
}
380385
}

0 commit comments

Comments
 (0)