Skip to content

Commit 6c4616b

Browse files
sarutakdongjoon-hyun
authored andcommitted
[SPARK-35990][BUILD] Remove avro-sbt plugin dependency
### What changes were proposed in this pull request? This PR removes sbt-avro plugin dependency. In the current master, Build with SBT depends on the plugin but it seems never used. Originally, the plugin was introduced for `flume-sink` in SPARK-1729 (#807) but `flume-sink` is no longer in Spark repository. After SBT was upgraded to 1.x in SPARK-21708 (#29286), `avroGenerate` part was introduced in `object SQL` in `SparkBuild.scala`. It's confusable but I understand `Test / avroGenerate := (Compile / avroGenerate).value` is for suppressing sbt-avro for `sql` sub-module. In fact, Test/compile will fail if `Test / avroGenerate :=(Compile / avroGenerate).value` is commented out. `sql` sub-module contains `parquet-compat.avpr` and `parquet-compat.avdl` but according to `sql/core/src/test/README.md`, they are intended to be handled by `gen-avro.sh`. Also, in terms of Maven build, there seems to be no definition to handle `*.avpr` or `*.avdl`. Based on the above, I think we can remove `sbt-avro`. ### Why are the changes needed? If `sbt-avro` is really no longer used, it's confusable that `sbt-avro` related configurations are in `SparkBuild.scala`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? GA. Closes #33190 from sarutak/remove-avro-from-sbt. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent a6e00ee commit 6c4616b

File tree

2 files changed

+1
-9
lines changed

2 files changed

+1
-9
lines changed

project/SparkBuild.scala

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -737,9 +737,6 @@ object Catalyst {
737737
}
738738

739739
object SQL {
740-
741-
import sbtavro.SbtAvro.autoImport._
742-
743740
lazy val settings = Seq(
744741
(console / initialCommands) :=
745742
"""
@@ -761,10 +758,8 @@ object SQL {
761758
|import sqlContext.implicits._
762759
|import sqlContext._
763760
""".stripMargin,
764-
(console / cleanupCommands) := "sc.stop()",
765-
Test / avroGenerate := (Compile / avroGenerate).value
761+
(console / cleanupCommands) := "sc.stop()"
766762
)
767-
768763
}
769764

770765
object Hive {

project/plugins.sbt

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,6 @@ addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.9.2")
3333

3434
addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.4.3")
3535

36-
addSbtPlugin("com.cavorite" % "sbt-avro" % "2.1.1")
37-
libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.10.2"
38-
3936
addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
4037

4138
libraryDependencies += "org.ow2.asm" % "asm" % "9.1"

0 commit comments

Comments
 (0)