SPARK-4628: Put all external projects behind a build flag

[pwendell]

This is important since we don't want breaks in the dependency
graphs of external projects to break the default Spark build.

[pwendell]

Jenkins, test this please.

[arahuja]

Should this be external-projects?

[pwendell]

yes - yes it should

[SparkQA]

Test build #23903 has started for PR 3485 at commit a1f575f.

• This patch merges cleanly.

[SparkQA]

Test build #23905 has started for PR 3485 at commit f3a3f13.

• This patch merges cleanly.

[sryza]

in less -> unless

[sryza]

external/kafka will still end up being built without the flag, right?

Also, will this not make the default build fail, because the examples depend on the external projects, and examples still get built automatically?

[SparkQA]

Test build #23903 has finished for PR 3485 at commit a1f575f.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23903/
Test FAILed.

[SparkQA]

Test build #23905 has finished for PR 3485 at commit f3a3f13.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23905/
Test PASSed.

[pwendell]

@sryza yeah I think you're right. Maybe we should as a more surgical fix just put the mqtt project behind a flag and for 1.3 we can clean it up properly (and maybe you could write this patch if you're interested).

[tdas]

@pwendell @sryza I agree with that as well. Lets just put mqtt behind the flag for now.

[tdas]

On second thought, does the current build of Spark break right now? If it does not, it may be fine to release Spark 1.2 as is.

[sryza]

I missed the context for why we would just put mqtt behind the flag. Something in its dependency graph is breaking the build?

[tdas]

@sryza I am trying to figure out the full extent of it on the current default build as well. The issue is the mqtt-client version 0.4.0 has been removed from certain repositories, which is causing breaks in the certain downstream systems. In case of Spark, I think it may cause builds of our previous versions to also break (havent tested myself). So this patch I think tries to make it more secure for the future, so that such build breaks in external systems do no break the building of core Spark.

@pwendell Please correct me if my understanding is incorrect.

[tdas]

I have made an alternative approach to upgrade MQTT from 0.4.0 to 1.0.1. See #3487 . I recommend going this route, rather than changing builds and stuff this late in the game.

[pwendell]

@sryza the issue is that mqtt-client is not hosted in maven central, it's maintained in an external repository owned by the eclipse project and also (for some versions) in a repository maintained by the Spring project. This is really a bad situation because we are relying on an artifact could simply be removed at some point in the future without warning and would break any Spark build.

For instance, all Spark 0.9.X builds are now broken because the eclipse repository removed mqtt-client 0.4.0. If in the future the spring repository also removes this (the spring repo was added to our build in Spark 1.0) many versions of Spark will fail to build.

As long as this is in the default build configuration we expose ourself to this, so I'm proposing removing mqtt from the default build. We should have a firm policy moving forward that we can't depend on artifacts that aren't hosted in maven central.

[srowen]

I think updating to MQTT 1.0.1 per @tdas is a good move. However @prabeesh seems to have found the new home of this old artifact (haven't tested it): #3492 Maybe that is a good PR to back-port, while moving ahead with MQTT 1.0.1 in the short term.

[srowen]

Oops, I think I spoke too soon. I actually just clicked through the paho-releases repo and ...

https://repo.eclipse.org/content/repositories/paho-releases/org/eclipse/paho/mqtt-client/0.4.0/

... is an empty directory.

[prabeesh]

@srowen they updated the repo

[pwendell]

That's great that they updated the repo. Nonetheless I think we should put this module behind a build flag for the reasons stated earlier. We can't be dependent on a third party repo for our default build to work, in general. So for 1.2 I'll just have an mqtt flag. For 1.3 we separately had thought of isolating these from the default build more generally, but we can defer that. So I'll close this patch and open up a simpler one.

[pwendell]

Actually even flagging only MQTT is complicated because of the examples. So I'll just punt on this until later. It just leaves us in a situation where we might have to make a patch release down the road if their repo changes, which isn't great.