Skip to content

Commit 01b4f38

Browse files
LuciferYangbeliefer
authored andcommitted
[SPARK-41206][SQL][FOLLOWUP] Make result of checkColumnNameDuplication stable to fix COLUMN_ALREADY_EXISTS check failed with Scala 2.13
### What changes were proposed in this pull request? This pr add a sort when `columnAlreadyExistsError` will be thrown to make the result of `SchemaUtils#checkColumnNameDuplication` stable. ### Why are the changes needed? Fix `COLUMN_ALREADY_EXISTS` check failed with Scala 2.13 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - Manual test: ``` dev/change-scala-version.sh 2.13 build/sbt clean "sql/testOnly org.apache.spark.sql.DataFrameSuite" -Pscala-2.13 build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV1Suite" -Pscala-2.13 build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV2Suite" -Pscala-2.13 build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonLegacyTimeParserSuite" -Pscala-2.13 ``` All tests passed Closes apache#38764 from LuciferYang/SPARK-41206. Authored-by: yangjie01 <[email protected]> Signed-off-by: Max Gekk <[email protected]>
1 parent 5e15c6d commit 01b4f38

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ private[spark] object SchemaUtils {
107107
val names = if (caseSensitiveAnalysis) columnNames else columnNames.map(_.toLowerCase)
108108
// scalastyle:on caselocale
109109
if (names.distinct.length != names.length) {
110-
val columnName = names.groupBy(identity).collectFirst {
110+
val columnName = names.groupBy(identity).toSeq.sortBy(_._1).collectFirst {
111111
case (x, ys) if ys.length > 1 => x
112112
}.get
113113
throw QueryCompilationErrors.columnAlreadyExistsError(columnName)

0 commit comments

Comments
 (0)