[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30451

AngersZhuuuu · 2020-11-21T04:31:47Z

What changes were proposed in this pull request?

We skip test HiveExternalCatalogVersionsSuite when testing with JAVA_9 or later because our previous version does not support JAVA_9 or later. We now add it back since we have a version supports JAVA_9 or later.

Why are the changes needed?

To recover test coverage.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Check CI logs.

AngersZhuuuu · 2020-11-21T04:32:26Z

FYI @dongjoon-hyun

SparkQA · 2020-11-21T05:19:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36065/

SparkQA · 2020-11-21T05:44:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36065/

SparkQA · 2020-11-21T06:31:53Z

Test build #131459 has finished for PR 30451 at commit 9400d9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GetShufflePushMergerLocations(numMergersNeeded: Int, hostsToFilter: Set[String])
case class RemoveShufflePushMergerLocation(host: String) extends ToBlockManagerMaster
abstract class LikeAllBase extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant
case class LikeAll(child: Expression, patterns: Seq[UTF8String]) extends LikeAllBase
case class NotLikeAll(child: Expression, patterns: Seq[UTF8String]) extends LikeAllBase
case class ParseUrl(children: Seq[Expression], failOnError: Boolean = SQLConf.get.ansiEnabled)

AngersZhuuuu · 2020-11-21T12:03:46Z

I found that after this change , it download zookeeper test-jar

downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar ...
2020-11-20 21:45:20.389 - stderr> 	[SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.4.6!zookeeper.jar(test-jar) (11ms)

But. right case is

downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.3/zookeeper-3.4.3.jar ...
2020-11-19 20:25:25.892 - stderr> 	[SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.4.3!zookeeper.jar (20ms)

See the logical of dowloadVersion

  def resolveDependencyPaths(
      artifacts: Array[AnyRef],
      cacheDirectory: File): String = {
    artifacts.map { ai =>
      val artifactInfo = ai.asInstanceOf[Artifact]
      val artifact = artifactInfo.getModuleRevisionId
      val testSuffix = if (artifactInfo.getType == "test-jar") "-tests" else ""
      cacheDirectory.getAbsolutePath + File.separator +
        s"${artifact.getOrganisation}_${artifact.getName}-${artifact.getRevision}${testSuffix}.jar"
    }.mkString(",")
  }

Seems some place artifact info is wrong?

SparkQA · 2020-11-21T13:11:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36070/

SparkQA · 2020-11-21T13:27:01Z

Test build #131465 has finished for PR 30451 at commit 053974a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-21T13:33:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36071/

SparkQA · 2020-11-21T13:42:42Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36070/

SparkQA · 2020-11-21T14:04:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36071/

SparkQA · 2020-11-21T14:31:50Z

Test build #131464 has finished for PR 30451 at commit 67222ca.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-21T15:24:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36076/

SparkQA · 2020-11-21T15:50:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36076/

dongjoon-hyun · 2020-11-21T16:23:19Z

Thank you for working on this, @AngersZhuuuu !

dongjoon-hyun · 2020-11-21T16:26:21Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala

@@ -103,6 +103,7 @@ package object client {
    // Since HIVE-14496, Hive materialized view need calcite-core.
    // For spark, only VersionsSuite currently creates a hive materialized view for testing.
    case object v2_3 extends HiveVersion("2.3.7",
+      extraDeps = Seq("org.apache.zookeeper:zookeeper:3.4.6"),


Although we have this, the GitHub Action is stilling failing. Maybe, we have other issues?

Although we have this, the GitHub Action is stilling failing. Maybe, we have other issues?

Yes, with zookeeper dependency but ivy still. download zookeeper with it's test jars, after detail check I can't find code issue related to this problem. so strange.

Although we have this, the GitHub Action is stilling failing. Maybe, we have other issues?

Since default hive version is 2.3.7 and 2.3.7 can pass jdk9+, so. I changed hive version. in HiveExternalCatalogVersionSuite. It's reasonable.

So confused, still working to find why it download zookeeper(test-jar).

In maven repo I noticed that only zokeeper-3.4.x have pom, but it doesn't seem like this is the problem

I agree with you that it's weird. Maybe, could you try this Spark Shell? If we starts without local caches (Maven/Ivy), I guess Spark Shell will fail with the same reason.

Then, we should fix it first as an independent JIRA issue.

I tried in feature #29966 to download zookeeper-3.4.6(after clean cache)

test("SPARK-33084: Add jar support ivy url") { sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local-cluster[3, 1, 1024]")) sc.addJar("ivy://org.apache.zookeeper:zookeeper:3.4.6") println("+===================+") sc.listJars().foreach(println) assert(sc.listJars().find(_.contains("zookeeper-3.4.6")).nonEmpty) }

It really download right jar.
But with UT

test("SPARK-33084: Add jar support ivy url") { sc = new SparkContext(new SparkConf().setAppName("test") .setMaster("local-cluster[3, 1, 1024]")) sc.addJar("ivy://org.apache.hive:hive-exec:2.3.7" + "?exclude=org.pentaho:pentaho-aggdesigner-algorithm&transitive=true") println("+===================+") sc.listJars().foreach(println) }

It download both zookeeper.jar and zookeeper-test.jar.

@dongjoon-hyun
This error happened when HiveExternalCatalogVersionsSuite.prepare(), So any change in current code won't work since it use downloaded compiled spark.
So change in current code won't change the result.

SparkQA · 2020-11-21T16:36:22Z

Test build #131470 has finished for PR 30451 at commit 668db8b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-21T18:24:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36082/

SparkQA · 2020-11-21T18:54:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36082/

SparkQA · 2020-11-21T19:36:03Z

Test build #131476 has finished for PR 30451 at commit b853f5e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-22T01:52:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36091/

AngersZhuuuu · 2020-11-22T04:20:21Z

@dongjoon-hyun
I tried download spark-3.0.1-hadoop-2.7 and start it with

./bin/spark-sql --conf spark.sql.hive.metastore.version=2.3.7 --conf spark.sql.hive.metastore.jars=maven

Will download zookeeper-3.4.6.jar and zookeeper-3.4.6-test.jar, and it works well.

After change hive version 1.2.1 when jdk < jdk9 and hive version 2.3.7 when jdk9+ can pass the GitAction.

SparkQA · 2020-11-22T04:24:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36094/

SparkQA · 2020-11-22T04:55:05Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36094/

SparkQA · 2020-11-22T04:55:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36095/

SparkQA · 2020-11-22T05:22:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36095/

SparkQA · 2020-11-22T05:28:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36096/

SparkQA · 2020-11-22T05:36:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36097/

SparkQA · 2020-11-22T05:37:07Z

Test build #131490 has finished for PR 30451 at commit ac2afc0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-22T05:59:40Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36096/

SparkQA · 2020-11-22T06:01:01Z

Test build #131493 has finished for PR 30451 at commit d21eb17.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-22T06:03:14Z

retest this please

SparkQA · 2020-11-22T06:03:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36097/

SparkQA · 2020-11-22T06:03:57Z

Test build #131491 has finished for PR 30451 at commit 4390fca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-22T06:15:19Z

Test build #131492 has finished for PR 30451 at commit d6f957c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-22T06:36:28Z

Test build #131494 has finished for PR 30451 at commit 568150a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-22T06:48:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36099/

SparkQA · 2020-11-22T06:50:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36098/

SparkQA · 2020-11-22T07:16:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36098/

SparkQA · 2020-11-22T07:21:01Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36099/

SparkQA · 2020-11-22T08:05:02Z

Test build #131495 has finished for PR 30451 at commit 568150a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-22T08:54:21Z

retest this please

SparkQA · 2020-11-22T09:38:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36102/

SparkQA · 2020-11-22T10:01:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36102/

SparkQA · 2020-11-22T10:56:44Z

Test build #131498 has finished for PR 30451 at commit 568150a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-11-22T18:29:53Z

Thank you so much, @AngersZhuuuu . Merged to master.

AngersZhuuuu added 3 commits November 19, 2020 22:28

Update HiveExternalCatalogVersionsSuite.scala

d553765

Update HiveExternalCatalogVersionsSuite.scala

b9dd1e4

Merge branch 'master' into SPARK-28704

9400d9a

github-actions bot added the SQL label Nov 21, 2020

AngersZhuuuu added 2 commits November 21, 2020 20:28

Update package.scala

67222ca

with extra Deps

053974a

Update package.scala

668db8b

dongjoon-hyun reviewed Nov 21, 2020

View reviewed changes

test

b853f5e

AngersZhuuuu added 2 commits November 22, 2020 09:10

hive version change with jdk version

c4543ef

Update IsolatedClientLoader.scala

568150a

github-actions bot added the INFRA label Nov 22, 2020

AngersZhuuuu force-pushed the SPARK-28704 branch from d21eb17 to 568150a Compare November 22, 2020 06:00

AngersZhuuuu changed the title ~~[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+~~ [SPARK-28704][SQL][TEST][test-java11] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ Nov 22, 2020

dongjoon-hyun changed the title ~~[SPARK-28704][SQL][TEST][test-java11] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+~~ [SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ Nov 22, 2020

dongjoon-hyun closed this in d7f4b2a Nov 22, 2020

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30451

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30451

Uh oh!

Conversation

AngersZhuuuu commented Nov 21, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

AngersZhuuuu commented Nov 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

dongjoon-hyun commented Nov 21, 2020

Uh oh!

dongjoon-hyun Nov 21, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 21, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 21, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 21, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 21, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Nov 22, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 22, 2020

Uh oh!

AngersZhuuuu commented Nov 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 22, 2020

Uh oh!

AngersZhuuuu commented Nov 21, 2020 •

edited

Loading

AngersZhuuuu Nov 21, 2020 •

edited

Loading

dongjoon-hyun Nov 21, 2020 •

edited

Loading

AngersZhuuuu commented Nov 22, 2020 •

edited

Loading