Skip to content

[SPARK-4237][BUILD] Fix MANIFEST.MF in maven assembly jar #3103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-4237][BUILD] Fix MANIFEST.MF in maven assembly jar #3103

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e81962b
move core dep to the first
scwf Nov 6, 2014
8332304
Merge branch 'master' of https://github.com/apache/spark into t
scwf Nov 6, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,7 @@
</properties>

<dependencies>
<!-- Promote Guava to compile scope in this module so it's included while shading. -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>compile</scope>
</dependency>
<!-- Note: place this first to ensure the proper MANIFEST.MF-->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the way to control MANIFEST.MF since this is ordering is not guaranteed to affect anything at all. You want to exclude MANIFEST.MF from deps entirely, probably.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @srowen, if we exclude MANIFEST.MF from deps entirely, there will be no MANIFEST.MF in assembly jar. I failed to find a way to add MANIFEST.MF in shade plugin(assembly jar), it seems we can only set manifestEntries but can not create a new MANIFEST.MF in shade plugin.
And another issue is that if we exclude MANIFEST.MF but set manifestEntries, then the excluding will be invalid(see #3103 (comment))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No MANIFEST.MF file isn't a problem per se, if we don't want it to contain anything anyway. Why not exclude MANIFEST.MF and also set entries you want? I don't see that option tried. but, the latter is really optional. You don't even need MANIFEST.MF at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So here i place core dep first and overwrite version info by setting manifestEntries, or any advice to make a clean MANIFEST.MF in shade plugin?
Another solution is to publish a empty jar with proper MANIFEST.MF for this issue, but i do not suggest that because it make this more complex.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ordering does not necessarily guarantee this result. Why do you want MANIFEST.MF at all? The discussion here suggests it is not and does not need to be used.

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
Expand Down Expand Up @@ -84,6 +79,12 @@
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<!-- Promote Guava to compile scope in this module so it's included while shading. -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>compile</scope>
</dependency>
</dependencies>

<build>
Expand Down Expand Up @@ -148,6 +149,17 @@
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Implementation-Vendor>org.apache.spark</Implementation-Vendor>
<Implementation-Title>spark-assembly</Implementation-Title>
<Implementation-Version>${project.version}</Implementation-Version>
<Implementation-Vendor-Id>org.apache.spark</Implementation-Vendor-Id>
<Specification-Vendor>org.apache.spark</Specification-Vendor>
<Specification-Title>spark-assembly</Specification-Title>
<Specification-Version>${project.version}</Specification-Version>
</manifestEntries>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/services/org.apache.hadoop.fs.FileSystem</resource>
Expand Down