SPARK-1022 [BUILD] Depend on Zookeeper from Kafka tests

[srowen]

Per discussion at #1751 I suggest that the more correct thing to do is to depend on Zookeeper from Kafka tests. The first commit does this.

The second commit reflects the explicit dependency on ZK in core, in order to make zookeeper.version do something. Another reasonable change would be to simply remove zookeeper.version to reflect that otherwise it does nothing.

[srowen]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1804. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18018/consoleFull

[SparkQA]

QA results for PR 1804:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18018/consoleFull

[SparkQA]

QA tests have started for PR 1804. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18019/consoleFull

[SparkQA]

QA results for PR 1804:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18019/consoleFull

[tdas]

Continuing the discussion in #1751, an alternative solution is to apply this explicit zookeeper dependency on only the spark-stremaing-kafka module (since immediately explicitly needs it) as opposed to the spark-core module (which may make it more convenient to control zookeeper version across all vendors builds).

[srowen]

@tdas also 'porting' my comment -- the problem is that right now, without the dependency declared in core, there is no way to control which version of ZK goes into Spark core. You get what Curator happens to depend on. Core already depends on ZK indirectly.

However, I see no reason that Spark cares about ZK per se. Vendors can control the ZK versions downstream if it maters. So it's probably less confusing to just remove zookeeper.version too. Right now it's overridden in the MapR profile, as if it does something, when it doesn't.

The explicit dependency in Kafka tests can directly specify a version of, say, 3.4.5 for its own purpose. That fully contains the ZK dependency issue and seems entirely correct to me. Updated PR coming...

[SparkQA]

QA tests have started for PR 1804 at commit eb92ed8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1804 at commit eb92ed8.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • In multiclass classification, all$2^`
  • public final class JavaDecisionTree

[srowen]

Bah, I give up on this. Not clear whether the test failure is a false positive. At this point, the point of the patch is really just "zookeeper.properties is not used, but the build acts like it thinks it is used", and that's minor. Somebody zap that someday...