[SPARK-4261][BUILD][SQL] Make right version info for beeline #3124

scwf · 2014-11-06T00:51:20Z

More detail discuss link to #3103
Build spark with maven, then use beeline to connect jdbc/odbc server, the output will be

JackydeMacBook-Pro:spark1 jackylee$ bin/beeline 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Beeline version ??? by Apache Hive

This PR make right version info for beeline by excluding hive-beeline from the assembly jar, and deliver it as a separate jar to "lib_managed/jars", a WIP tag here is because i have not tested this locally.

scwf · 2014-11-06T00:52:39Z

More detail discuss link to #3103

SparkQA · 2014-11-06T00:55:12Z

Test build #22969 has started for PR 3124 at commit c19cd28.

This patch merges cleanly.

SparkQA · 2014-11-06T02:43:29Z

Test build #22969 has finished for PR 3124 at commit c19cd28.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T02:43:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22969/
Test PASSed.

scwf · 2014-11-06T04:56:21Z

sql/hive-thriftserver/pom.xml

@@ -95,6 +95,30 @@
          <skip>true</skip>
        </configuration>
      </plugin>
+      <!-- Deploy datanucleus jars to the spark/lib_managed/jars directory -->


SparkQA · 2014-11-06T05:59:55Z

Test build #22989 has started for PR 3124 at commit 8011701.

This patch merges cleanly.

scwf · 2014-11-06T06:00:02Z

Updated and locally test ok. With this PR the output will be

JackydeMacBook-Pro:spark jackylee$ bin/beeline 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Beeline version 0.13.1a by Apache Hive

srowen · 2014-11-06T07:45:21Z

sql/hive-thriftserver/pom.xml

+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-dependency-plugin</artifactId>
+        <version>2.4</version>


Nit: you should just inherit version config from the parent pom

Seems there is no maven-dependency-plugin version in parent Pom? @srowen

Oh, hm. I suggest adding it in <pluginManagement> -- just with the version. Also, use 2.9, not 2.4.

SparkQA · 2014-11-06T07:58:57Z

Test build #22989 has finished for PR 3124 at commit 8011701.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T07:59:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22989/
Test PASSed.

scwf · 2014-11-06T08:57:23Z

Updated.

SparkQA · 2014-11-06T08:59:57Z

Test build #22995 has started for PR 3124 at commit 4230322.

This patch merges cleanly.

SparkQA · 2014-11-06T10:47:17Z

Test build #22995 has finished for PR 3124 at commit 4230322.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T10:47:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22995/
Test PASSed.

scwf · 2014-11-06T19:48:56Z

@pwendell, can you take a look at this?

SparkQA · 2014-11-07T04:29:54Z

Test build #23035 has started for PR 3124 at commit 204bab1.

This patch merges cleanly.

SparkQA · 2014-11-07T06:16:58Z

Test build #23035 has finished for PR 3124 at commit 204bab1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-07T06:17:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23035/
Test PASSed.

AmplabJenkins · 2014-11-07T22:43:43Z

Can one of the admins verify this patch?

AmplabJenkins · 2014-11-07T22:52:33Z

Can one of the admins verify this patch?

scwf · 2014-11-08T18:20:11Z

cc @pwendell @marmbrus

marmbrus · 2014-11-14T21:54:07Z

@pwendell any thoughts on the build changes here?

scwf · 2014-11-18T16:10:22Z

ping @pwendell

scwf · 2014-11-26T05:10:14Z

ping

scwf · 2014-12-03T03:03:24Z

ping

pwendell · 2014-12-03T03:11:12Z

Is Hive reading the version from a manifest somewhere? It's sort of unfortunate to have to increase the number of special case jars for a cosmetic change like this. Can we instead just add the relevant manifest file to our assembly?

scwf · 2014-12-03T03:55:35Z

Hi @pwendell, Beeline relies on the Implementation-Version field in the manifest to inspect version info. And my other PR fixed the manifest issue(#3103) of maven building, after that PR the manifest of assembly jar is as follows:

Manifest-Version: 1.0
Implementation-Vendor: org.apache.spark
Implementation-Title: spark-assembly
Implementation-Version: 1.2.0-SNAPSHOT
Implementation-Vendor-Id: org.apache.spark
Specification-Vendor: org.apache.spark
Specification-Title: spark-assembly
Specification-Version: 1.2.0-SNAPSHOT

which is same with manifest of SBT assembly jar.

The problem of using assembly jar is that Hive beeline version will be 1.2.0-SNAPSHOT since it read the manifest file of assembly jar. But we need the correct version of hive beeline, which should be 0.13.1a, so here i exclud hive-beeline from the assembly jar.

We can not read the right version if using assembly jar

SparkQA · 2014-12-12T13:55:28Z

Test build #24409 has started for PR 3124 at commit 45ddd5b.

This patch merges cleanly.

SparkQA · 2014-12-12T15:39:08Z

Test build #24409 has finished for PR 3124 at commit 45ddd5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-12T15:39:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24409/
Test PASSed.

scwf · 2014-12-30T23:28:56Z

Ping

scwf · 2015-01-08T20:40:55Z

Ping

srowen · 2015-02-10T10:28:02Z

I also disagree with this change, as it seems to be saying we should declare Spark's version in the Manifest to be incorrect, and match Beeline's, just to fix a cosmetic issue.

scwf · 2015-02-10T10:46:29Z

ok， I am closing this since this issue do not affect functional and as @srowen said it is a "cosmetic issue"

make right version info for beeline

c19cd28

scwf changed the title ~~[WIP][SPARK-4261] Make right version info for beeline~~ [WIP][SPARK-4261][BUILD][SQL] Make right version info for beeline Nov 6, 2014

scwf reviewed Nov 6, 2014
View reviewed changes

bug fix

8011701

scwf changed the title ~~[WIP][SPARK-4261][BUILD][SQL] Make right version info for beeline~~ [SPARK-4261][BUILD][SQL] Make right version info for beeline Nov 6, 2014

scwf changed the title ~~[SPARK-4261][BUILD][SQL] Make right version info for beeline~~ [SPARK-4261][BUILD][SQL] Make right version info for beeline when building with maven Nov 6, 2014

srowen reviewed Nov 6, 2014
View reviewed changes

scwf mentioned this pull request Nov 6, 2014

[SPARK-4237][BUILD] Fix MANIFEST.MF in maven assembly jar #3103

Closed

inherit version config from the parent pom

4230322

fix beeline version when sbt

204bab1

scwf changed the title ~~[SPARK-4261][BUILD][SQL] Make right version info for beeline when building with maven~~ [SPARK-4261][BUILD][SQL] Make right version info for beeline Nov 7, 2014

fix conflicts

45ddd5b

scwf closed this Feb 10, 2015

[SPARK-4261][BUILD][SQL] Make right version info for beeline #3124

[SPARK-4261][BUILD][SQL] Make right version info for beeline #3124

Uh oh!

Conversation

scwf commented Nov 6, 2014

Uh oh!

scwf commented Nov 6, 2014

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

AmplabJenkins commented Nov 6, 2014

Uh oh!

scwf Nov 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

scwf commented Nov 6, 2014

Uh oh!

srowen Nov 6, 2014

Choose a reason for hiding this comment

Uh oh!

scwf Nov 6, 2014

Choose a reason for hiding this comment

Uh oh!

srowen Nov 6, 2014

Choose a reason for hiding this comment

Uh oh!

scwf Nov 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

AmplabJenkins commented Nov 6, 2014

Uh oh!

scwf commented Nov 6, 2014

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

AmplabJenkins commented Nov 6, 2014

Uh oh!

scwf commented Nov 6, 2014

Uh oh!

SparkQA commented Nov 7, 2014

Uh oh!

SparkQA commented Nov 7, 2014

Uh oh!

AmplabJenkins commented Nov 7, 2014

Uh oh!

AmplabJenkins commented Nov 7, 2014

Uh oh!

AmplabJenkins commented Nov 7, 2014

Uh oh!

scwf commented Nov 8, 2014

Uh oh!

marmbrus commented Nov 14, 2014

Uh oh!

scwf commented Nov 18, 2014

Uh oh!

scwf commented Nov 26, 2014

Uh oh!

scwf commented Dec 3, 2014

Uh oh!

pwendell commented Dec 3, 2014

Uh oh!

scwf commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

AmplabJenkins commented Dec 12, 2014

Uh oh!

scwf commented Dec 30, 2014

Uh oh!

scwf commented Jan 8, 2015