[SPARK-4047] - Generate runtime warnings for example implementation of PageRank

[varadharajan]

Based on SPARK-2434, this PR generates runtime warnings for example implementations (Python, Scala) of PageRank.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Are there other examples that should have the same warning? I think there are many more than this.

[varadharajan]

Here are list of scala examples that i think is similar / naive implementation of algorithms from MLlib or graphx.

1. LocalALS
2. LocalFileLR
3. LocalKMeans
4. LocalLR
5. SparkALS
6. SparkHdfsLR
7. SparkKMeans
8. SparkLR
9. SparkPageRank (*)
10. SparkTachyonHdfsLR (*)

Python examples:

1. ALS
2. kmeans
3. logistic_regression
4. pagerank (*)

Java examples:

1. JavaHdfsLR (*)
2. JavaPageRank (*)

(*) - Examples with missing warnings. I've updated JIRA with these details and also added warning for them

I've also corrected class names of existing LR examples. They were pointing to org.apache.spark.mllib.classification.LogisticRegression instead of org.apache.spark.mllib.classification.LogisticRegressionModel

I've excluded examples that compute transitive closures on graphs because i'm was not able to find corresponding implementations in graphx. Please let me know if i'm missing something

[davies]

Jenkins, ok to test.

[SparkQA]

Test build #22498 has started for PR 2894 at commit 252f595.

• This patch merges cleanly.

[SparkQA]

Test build #22498 has finished for PR 2894 at commit 252f595.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22498/
Test PASSed.

[davies]

LGTM, thanks!

[varadharajan]

Thanks :)

[JoshRosen]

Since this is MLlib related, @mengxr or @jkbradley, could one of you do the final sign-off + commit on this? Thanks!

[jkbradley]

@varadharajan Thanks for adding the warnings! My main comment is that LogisticRegressionModel is a model, rather than an algorithm. Users would really want the algorithm which they can run to produce the model. Could you instead direct users to the algorithms: LogisticRegressionWithSGD and LogisticRegressionWithLBFGS? (It is awkward that there are 2 algorithms to direct users towards, but it is hard to get around that.)

[varadharajan]

@jkbradley Makes sense. I've updated the warnings, please let me know if wordings can be improved. Also i just noticed that pyspark classification model does not have LR-LBFGS implementation. I'll probably create a new issue and work on it.

[SparkQA]

Test build #23102 has started for PR 2894 at commit 5f9406b.

• This patch merges cleanly.

[varadharajan]

Also i think it would help users if we can document in the LR section of the MLlib guide, which algorithm should be preferred in which scenarios.

[SparkQA]

Test build #23102 has finished for PR 2894 at commit 5f9406b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23102/
Test PASSed.

[jkbradley]

@varadharajan Good suggestion about documenting algs for LR; I'll make a note to do that for the upcoming release. Thank you for the PR!

LGTM

[varadharajan]

@jkbradley Thanks :)

[mengxr]

Merged into master and branch-1.2. Thanks! (We should find some time and clean really old examples.)