[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30428

AngersZhuuuu · 2020-11-19T14:31:08Z

What changes were proposed in this pull request?

We skip test HiveExternalCatalogVersionsSuite when testing with JAVA_9 or later because our previous version does not support JAVA_9 or later. We now add it back since we have a version supports JAVA_9 or later.

Why are the changes needed?

To recover test coverage.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Check CI logs.

SparkQA · 2020-11-19T15:19:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35958/

dongjoon-hyun · 2020-11-19T15:31:42Z

Thank you for this efforts, @AngersZhuuuu !

SparkQA · 2020-11-19T15:46:23Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35958/

SparkQA · 2020-11-19T17:03:20Z

Test build #131355 has finished for PR 30428 at commit d553765.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-11-19T20:45:22Z

Retest this please.

dongjoon-hyun · 2020-11-19T20:47:10Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

-    versions.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
+    versions
+      .filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
+      .filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_11))


Shall we use JavaVersion.JAVA_9 to be consistent with the PR title ~~~ at JDK9+?

dongjoon-hyun · 2020-11-19T20:49:25Z

BTW, @AngersZhuuuu . The title is correct? Please revise it according to the PR content.

Skip HiveExternalCatalogVersionsSuite and 3 tests in HiveSparkSubmitSuite at JDK9+

SparkQA · 2020-11-19T21:31:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35974/

SparkQA · 2020-11-19T21:54:35Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35974/

SparkQA · 2020-11-19T21:59:26Z

Test build #131370 has finished for PR 30428 at commit d553765.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-19T22:34:49Z

Test build #131371 has finished for PR 30428 at commit d553765.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-11-20T00:37:10Z

Could you take a look at HiveExternalCatalogVersionsSuite failure?

SparkQA · 2020-11-20T02:11:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35984/

SparkQA · 2020-11-20T02:35:37Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35984/

SparkQA · 2020-11-20T03:16:44Z

Test build #131381 has finished for PR 30428 at commit b9dd1e4.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-20T04:08:38Z

retest this please

AngersZhuuuu · 2020-11-20T04:09:09Z

Could you take a look at HiveExternalCatalogVersionsSuite failure?

updated, seems latest UT failed not this issue?

SparkQA · 2020-11-20T05:33:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36002/

SparkQA · 2020-11-20T05:56:44Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36002/

SparkQA · 2020-11-20T06:24:37Z

Test build #131398 has finished for PR 30428 at commit b9dd1e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-20T06:25:45Z

retest this please

SparkQA · 2020-11-20T07:07:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36005/

SparkQA · 2020-11-20T07:38:29Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36005/

SparkQA · 2020-11-20T08:05:02Z

Test build #131401 has finished for PR 30428 at commit b9dd1e4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-20T08:06:58Z

retest this please

SparkQA · 2020-11-20T08:53:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36011/

SparkQA · 2020-11-20T09:28:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36011/

SparkQA · 2020-11-20T10:18:52Z

Test build #131406 has finished for PR 30428 at commit b9dd1e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-20T10:24:08Z

ping @dongjoon-hyun All pass now.

dongjoon-hyun

+1, LGTM. Thank you so much, @AngersZhuuuu .
Merged to master for Apache Spark 3.1.

dongjoon-hyun · 2020-11-21T03:01:02Z

Ur, it seems that GitHub Action failed with java.io.FileNotFoundException: Source '/home/runner/.ivy2/jars/org.apache.zookeeper_zookeeper-3.4.6.jar'

2020-11-20T21:46:53.3518362Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> : java.io.FileNotFoundException: Source '/home/runner/.ivy2/jars/org.apache.zookeeper_zookeeper-3.4.6.jar' does not exist�[0m
2020-11-20T21:46:53.3520542Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1074)�[0m
2020-11-20T21:46:53.3522853Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:1013)�[0m
2020-11-20T21:46:53.3525435Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:977)�[0m
2020-11-20T21:46:53.3528409Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.spark.sql.hive.client.IsolatedClientLoader$.$anonfun$downloadVersion$4(IsolatedClientLoader.scala:139)�[0m
2020-11-20T21:46:53.3531793Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at

dongjoon-hyun · 2020-11-21T03:01:43Z

And, it seems to happen in a clean installation setup like GitHub Action. I'll revert this quickly. Sorry, @AngersZhuuuu .

AngersZhuuuu · 2020-11-21T03:32:48Z

Ur, it seems that GitHub Action failed with java.io.FileNotFoundException: Source '/home/runner/.ivy2/jars/org.apache.zookeeper_zookeeper-3.4.6.jar'

2020-11-20T21:46:53.3518362Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> : java.io.FileNotFoundException: Source '/home/runner/.ivy2/jars/org.apache.zookeeper_zookeeper-3.4.6.jar' does not exist�[0m
2020-11-20T21:46:53.3520542Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1074)�[0m
2020-11-20T21:46:53.3522853Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:1013)�[0m
2020-11-20T21:46:53.3525435Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:977)�[0m
2020-11-20T21:46:53.3528409Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at org.apache.spark.sql.hive.client.IsolatedClientLoader$.$anonfun$downloadVersion$4(IsolatedClientLoader.scala:139)�[0m
2020-11-20T21:46:53.3531793Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  2020-11-20 13:46:51.755 - stdout> 	at

In the build information it download zookeeper successfully. Ur a little strange, what should I need to do next?

AngersZhuuuu · 2020-11-21T03:33:40Z

And, it seems to happen in a clean installation setup like GitHub Action. I'll revert this quickly. Sorry, @AngersZhuuuu .

It's ok, do I need to open a new PR to test GitAction？

dongjoon-hyun · 2020-11-21T04:03:32Z

Yes. Please, @AngersZhuuuu . We can focus on the GitHub Action result only.

Update HiveExternalCatalogVersionsSuite.scala

d553765

AngersZhuuuu marked this pull request as draft November 19, 2020 14:31

github-actions bot added the SQL label Nov 19, 2020

dongjoon-hyun changed the title ~~[SPARK-28704][SQL][TEST] Skip HiveExternalCatalogVersionsSuite and 3 tests in HiveSparkSubmitSuite at JDK9+~~ [SPARK-28704][SQL][TEST][test-java11] Skip HiveExternalCatalogVersionsSuite and 3 tests in HiveSparkSubmitSuite at JDK9+ Nov 19, 2020

dongjoon-hyun reviewed Nov 19, 2020

View reviewed changes

Update HiveExternalCatalogVersionsSuite.scala

b9dd1e4

AngersZhuuuu changed the title ~~[SPARK-28704][SQL][TEST][test-java11] Skip HiveExternalCatalogVersionsSuite and 3 tests in HiveSparkSubmitSuite at JDK9+~~ [SPARK-28704][SQL][TEST][test-java11] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ Nov 20, 2020

AngersZhuuuu changed the title ~~[SPARK-28704][SQL][TEST][test-java11] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+~~ [SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ Nov 20, 2020

dongjoon-hyun marked this pull request as ready for review November 20, 2020 16:38

dongjoon-hyun approved these changes Nov 20, 2020

View reviewed changes

dongjoon-hyun closed this in 47326ac Nov 20, 2020

AngersZhuuuu deleted the SPARK-28704 branch November 21, 2020 04:12

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30428

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30428

Conversation

AngersZhuuuu commented Nov 19, 2020 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

dongjoon-hyun Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

AngersZhuuuu commented Nov 20, 2020

Uh oh!

AngersZhuuuu commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

AngersZhuuuu commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

AngersZhuuuu commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

AngersZhuuuu commented Nov 20, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 21, 2020

Uh oh!

dongjoon-hyun commented Nov 21, 2020

Uh oh!

AngersZhuuuu commented Nov 21, 2020

Uh oh!

AngersZhuuuu commented Nov 19, 2020 •

edited by dongjoon-hyun

Loading

dongjoon-hyun Nov 19, 2020 •

edited

Loading

dongjoon-hyun commented Nov 19, 2020 •

edited

Loading