[SPARK-26030][BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0 #22977

cloud-fan · 2018-11-08T11:37:41Z

What changes were proposed in this pull request?

Since Spark 2.4.0 is already in maven repo, we can Bump previousSparkVersion in MimaBuild.scala to be 2.4.0.

Note that, seems we forgot to do it for branch 2.4, so this PR also updates MimaExcludes.scala

How was this patch tested?

N/A

HyukjinKwon

Looks good if tests pass

cloud-fan · 2018-11-08T11:43:12Z

I think one major issue is, there is no document about how to update mima with new releases. Anyone knows the detailed process? Seems we need to update MimaExcludes.scala with something like lazy val v30excludes = v24excludes ++ Seq when cut the new branch, and update MimaBuild.scala when the new release is published.

And there are 2 remaining issues.

the data source v2 changes broke a lot of mima rules, while I expect interfaces marked as "Envolving" should not be tracked by mima.
mllib broke a lot of mima rules.

Also cc @JoshRosen @srowen @gatorsmile @shaneknapp @vanzin @holdenk @felixcheung

HyukjinKwon · 2018-11-08T11:48:21Z

Actually, similar changes were being made at #22967 FYI. cc @dbtsai

SparkQA · 2018-11-08T11:52:02Z

Test build #98596 has finished for PR 22977 at commit a035778.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-08T11:56:29Z

project/MimaBuild.scala

@@ -88,7 +88,7 @@ object MimaBuild {

  def mimaSettings(sparkHome: File, projectRef: ProjectRef) = {
    val organization = "org.apache.spark"
-    val previousSparkVersion = "2.2.0"
+    val previousSparkVersion = "2.4.0"


According to https://issues.apache.org/jira/browse/SPARK-23070, 2.3.0 seems to be the correct version in this PR. (If this is a part of 2.4.0 release process)

cc @gatorsmile .

up to my understanding, we should have changed it to 2.3.0 when 2.3.0 was released.

Maybe we should send another PR to branch-2.4 and change it to 2.3.0

I believe @cloud-fan is right here. If master is 3.0, the previous version is 2.4.0 in master. Analogously for branch 2.4. Yes sounds like this should be a step in the release process somewhere.

Thanks, @cloud-fan and @srowen . +1 for another PR for branch-2.4.

cloud-fan · 2018-11-08T12:44:36Z

project/MimaExcludes.scala

+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.Window.rangeBetween"),
+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"),
+    // [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager
+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"),


This is actually a private method, I'm not sure why mima tracks it.

It was a private[spark] method which has to be implemented as a public method in the Java bytecode. MiMa doesn't know the difference. Yes it can be ignore for sure.

So these failures are 'new' because they were introduced after Spark 2.2 (hence not missing since 2.2) then removed, so bumping the previous version causes them to fail now? OK.

srowen · 2018-11-08T15:33:02Z

project/MimaBuild.scala

@@ -88,7 +88,7 @@ object MimaBuild {

  def mimaSettings(sparkHome: File, projectRef: ProjectRef) = {
    val organization = "org.apache.spark"
-    val previousSparkVersion = "2.2.0"
+    val previousSparkVersion = "2.4.0"


I believe @cloud-fan is right here. If master is 3.0, the previous version is 2.4.0 in master. Analogously for branch 2.4. Yes sounds like this should be a step in the release process somewhere.

srowen · 2018-11-08T15:34:40Z

project/MimaExcludes.scala

+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.Window.rangeBetween"),
+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"),
+    // [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager
+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"),


It was a private[spark] method which has to be implemented as a public method in the Java bytecode. MiMa doesn't know the difference. Yes it can be ignore for sure.

So these failures are 'new' because they were introduced after Spark 2.2 (hence not missing since 2.2) then removed, so bumping the previous version causes them to fail now? OK.

srowen · 2018-11-10T15:55:40Z

project/MimaExcludes.scala

+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"),
+    // [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager
+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"),
+    // [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3


@cloud-fan really these two lines should move to the top, because the first long chunk of exclusions is for 25908.

felixcheung · 2018-11-10T20:44:09Z

I think also there is a hive metastore test that downloads spark release jar?

srowen · 2018-11-10T20:55:04Z

HiveExternalCatalogVersionsSuite ? that's already updated.

felixcheung · 2018-11-11T07:55:19Z

right, I mean both this and that should be part of the process "post-release"
about @cloud-fan asks #22977 (comment)

cloud-fan · 2018-11-13T03:22:49Z

project/MimaExcludes.scala

-    ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.classification.LabelConverter")
+    ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.classification.LabelConverter"),
+
+    // [SPARK-21842][MESOS] Support Kerberos ticket renewal and creation in Mesos


these changes are cherry-picked from #23015

SparkQA · 2018-11-13T03:37:50Z

Test build #98746 has finished for PR 22977 at commit 8b9efe1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

…a to be 2.3.0 ## What changes were proposed in this pull request? Although it's a little late, we should still update mima for branch 2.4, to avoid future breaking changes. Note that, when merging, we should forward port it to master branch, so that the excluding rules are still in `v24excludes`. TODO: update the release process document to mention about mima update. ## How was this patch tested? N/A Closes apache#23015 from cloud-fan/mima-2.4. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

SparkQA · 2018-11-13T04:33:01Z

Test build #98750 has finished for PR 22977 at commit 8d9f5c7.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

dbtsai · 2018-11-13T06:12:28Z

LGTM. Thanks!

cloud-fan · 2018-11-13T06:14:16Z

since this PR only touches mima, and the jenkins already passed the mima check, I'm going to merge it to master, thanks!

SparkQA · 2018-11-13T08:05:02Z

Test build #98753 has finished for PR 22977 at commit 802b521.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

…be 2.4.0 ## What changes were proposed in this pull request? Since Spark 2.4.0 is already in maven repo, we can Bump previousSparkVersion in MimaBuild.scala to be 2.4.0. Note that, seems we forgot to do it for branch 2.4, so this PR also updates MimaExcludes.scala ## How was this patch tested? N/A Closes apache#22977 from cloud-fan/mima. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

HyukjinKwon approved these changes Nov 8, 2018

View reviewed changes

dongjoon-hyun reviewed Nov 8, 2018

View reviewed changes

cloud-fan commented Nov 8, 2018

View reviewed changes

srowen approved these changes Nov 8, 2018

View reviewed changes

srowen reviewed Nov 10, 2018

View reviewed changes

dbtsai mentioned this pull request Nov 11, 2018

[SPARK-25956] Make Scala 2.12 as default Scala version in Spark 3.0 #22967

Closed

cloud-fan force-pushed the mima branch from a035778 to 8b9efe1 Compare November 13, 2018 03:21

cloud-fan changed the title ~~[BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0~~ [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0 Nov 13, 2018

cloud-fan commented Nov 13, 2018

View reviewed changes

cloud-fan force-pushed the mima branch from 8b9efe1 to 8d9f5c7 Compare November 13, 2018 04:17

Bump previousSparkVersion in MimaBuild.scala to be 2.4.0

802b521

cloud-fan force-pushed the mima branch from 8d9f5c7 to 802b521 Compare November 13, 2018 04:56

dongjoon-hyun approved these changes Nov 13, 2018

View reviewed changes

asfgit closed this in e25bce5 Nov 13, 2018

LorenzoMartini mentioned this pull request Apr 19, 2021

[SPARK-21708][BUILD] Migrate build to sbt 1.x palantir/spark#759

Merged

[SPARK-26030][BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0 #22977

[SPARK-26030][BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0 #22977

Uh oh!

Conversation

cloud-fan commented Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Nov 8, 2018

Uh oh!

SparkQA commented Nov 8, 2018

Uh oh!

dongjoon-hyun Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Nov 10, 2018 via email

Uh oh!

srowen commented Nov 10, 2018

Uh oh!

felixcheung commented Nov 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 13, 2018

Uh oh!

SparkQA commented Nov 13, 2018

Uh oh!

dbtsai commented Nov 13, 2018

Uh oh!

cloud-fan commented Nov 13, 2018

Uh oh!

SparkQA commented Nov 13, 2018

Uh oh!

Uh oh!

cloud-fan commented Nov 8, 2018 •

edited

Loading

cloud-fan commented Nov 8, 2018 •

edited

Loading

dongjoon-hyun Nov 8, 2018 •

edited

Loading

cloud-fan Nov 8, 2018 •

edited

Loading

felixcheung commented Nov 11, 2018 •

edited

Loading