[SPARK-6806] [SparkR] [Docs] Fill in SparkR examples in programming guide #5442

davies · 2015-04-09T22:23:24Z

sqlCtx -> sqlContext

You can check the docs by:

$ cd docs
$ SKIP_SCALADOC=1 jekyll serve

cc @shivaram

sqlCtx -> sqlContext

SparkQA · 2015-04-09T22:28:21Z

Test build #29975 has started for PR 5442 at commit 23f751a.

SparkQA · 2015-04-09T22:48:42Z

Test build #29979 has started for PR 5442 at commit 9c2a062.

shivaram · 2015-04-09T22:49:53Z

@cafreeman -- If you get a chance, could you take a look at this too ?

SparkQA · 2015-04-09T23:50:37Z

Test build #29975 has finished for PR 5442 at commit 23f751a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-09T23:50:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29975/
Test PASSed.

SparkQA · 2015-04-10T00:34:30Z

Test build #29979 has finished for PR 5442 at commit 9c2a062.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-10T00:34:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29979/
Test PASSed.

cafreeman · 2015-04-10T01:05:30Z

docs/programming-guide.md

+context connects to using the `--master` argument. You can also add dependencies
+(e.g. Spark Packages) to your shell session by supplying a comma-separated list of maven coordinates
+to the `--packages` argument. Any additional repositories where dependencies might exist (e.g. SonaType)
+can be passed to the `--repositories` argument. For example, to run `bin/pyspark` on exactly four cores, use:


This should refer to SparkR instead of PySpark.

cafreeman · 2015-04-10T01:34:57Z

Left some comments inline, but most of them seem like minor details leftover from translating the PySpark docs. Overall I think this is looking really good.

shivaram · 2015-04-10T01:45:16Z

docs/index.md

@@ -54,6 +54,15 @@ Example applications are also provided in Python. For example,

    ./bin/spark-submit examples/src/main/python/pi.py 10

+Spark also provides a R API. To run Spark interactively in a R interpreter, use


I think here (or somewhere close by) we should say that SparkR is an experimental component in <SPARK_VERSION> and that only the RDD API and DataFrame APIs have been implemented in SparkR.

AmplabJenkins · 2015-04-10T21:07:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30047/
Test FAILed.

SparkQA · 2015-04-10T21:09:50Z

Test build #660 has started for PR 5442 at commit 2f10a77.

SparkQA · 2015-04-10T21:18:36Z

Test build #30052 has started for PR 5442 at commit 3ef7cf3.

SparkQA · 2015-04-10T22:36:13Z

Test build #660 has finished for PR 5442 at commit 2f10a77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-10T22:43:48Z

Test build #30052 has finished for PR 5442 at commit 3ef7cf3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-10T22:43:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30052/
Test PASSed.

shivaram · 2015-04-10T22:58:25Z

R/pkg/R/pairRDD.R

@@ -327,7 +327,7 @@ setMethod("reduceByKey",
              convertEnvsToList(keys, vals)
            }
            locallyReduced <- lapplyPartition(x, reduceVals)
-            shuffled <- partitionBy(locallyReduced, numPartitions)
+            shuffled <- partitionBy(locallyReduced, as.integer(numPartitions))


We should use numToInt from utils.R here

AmplabJenkins · 2015-05-17T09:31:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32931/
Test PASSed.

shivaram · 2015-05-18T05:15:33Z

docs/sql-programming-guide.md

@@ -491,6 +573,37 @@ for teenName in teenNames.collect():

 </div>

+<div data-lang="r"  markdown="1">
+


This is not applicable right now

AmplabJenkins · 2015-05-18T15:52:11Z

Merged build triggered.

AmplabJenkins · 2015-05-18T15:52:18Z

Merged build started.

SparkQA · 2015-05-18T15:54:27Z

Test build #32998 has started for PR 5442 at commit 8496b26.

SparkQA · 2015-05-18T17:31:27Z

Test build #32998 has finished for PR 5442 at commit 8496b26.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T17:31:33Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-18T17:31:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32998/
Test FAILed.

SparkQA · 2015-05-18T20:11:32Z

Test build #821 has started for PR 5442 at commit 8496b26.

SparkQA · 2015-05-18T20:17:16Z

Test build #821 has finished for PR 5442 at commit 8496b26.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-19T00:42:40Z

Test build #825 has started for PR 5442 at commit 8496b26.

SparkQA · 2015-05-19T02:30:21Z

Test build #825 has finished for PR 5442 at commit 8496b26.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-19T22:12:13Z

Merged build triggered.

AmplabJenkins · 2015-05-19T22:12:19Z

Merged build started.

SparkQA · 2015-05-19T22:13:25Z

Test build #33103 has started for PR 5442 at commit 7a12ec6.

SparkQA · 2015-05-19T23:23:59Z

Test build #33103 has finished for PR 5442 at commit 7a12ec6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-19T23:24:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-19T23:24:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33103/
Test FAILed.

davies · 2015-05-21T23:36:42Z

@shivaram Is this ready to go?

SparkQA · 2015-05-21T23:36:53Z

Test build #850 has started for PR 5442 at commit 7a12ec6.

SparkQA · 2015-05-22T00:50:11Z

Test build #850 has finished for PR 5442 at commit 7a12ec6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shivaram · 2015-05-23T06:49:51Z

docs/sql-programming-guide.md

+
+{% highlight r %}
+
+df <- laodDF(sqlContext, source="jdbc", url="jdbc:postgresql:dbserver", dbtable="schema.tablename")


Minor typo: This should be loadDF

shivaram · 2015-05-23T06:50:51Z

@davies Sorry for the delay in looking at this. I think this change looks pretty good -- I found a minor typo that we can fix up during merge.

I think it might be better to actually create a new page for SparkR rather than append it to the DataFrames page -- but I'l do this in a follow PR.

LGTM

@shivaram

…uide sqlCtx -> sqlContext You can check the docs by: ``` $ cd docs $ SKIP_SCALADOC=1 jekyll serve ``` cc shivaram Author: Davies Liu <[email protected]> Closes #5442 from davies/r_docs and squashes the following commits: 7a12ec6 [Davies Liu] remove rdd in R docs 8496b26 [Davies Liu] remove the docs related to RDD e23b9d6 [Davies Liu] delete R docs for RDD API 222e4ff [Davies Liu] Merge branch 'master' into r_docs 89684ce [Davies Liu] Merge branch 'r_docs' of github.com:davies/spark into r_docs f0a10e1 [Davies Liu] address comments from @shivaram f61de71 [Davies Liu] Update pairRDD.R 3ef7cf3 [Davies Liu] use + instead of function(a,b) a+b 2f10a77 [Davies Liu] address comments from @cafreeman 9c2a062 [Davies Liu] mention R api together with Python API 23f751a [Davies Liu] Fill in SparkR examples in programming guide (cherry picked from commit 7af3818) Signed-off-by: Shivaram Venkataraman <[email protected]>

@shivaram

…uide sqlCtx -> sqlContext You can check the docs by: ``` $ cd docs $ SKIP_SCALADOC=1 jekyll serve ``` cc shivaram Author: Davies Liu <[email protected]> Closes apache#5442 from davies/r_docs and squashes the following commits: 7a12ec6 [Davies Liu] remove rdd in R docs 8496b26 [Davies Liu] remove the docs related to RDD e23b9d6 [Davies Liu] delete R docs for RDD API 222e4ff [Davies Liu] Merge branch 'master' into r_docs 89684ce [Davies Liu] Merge branch 'r_docs' of github.com:davies/spark into r_docs f0a10e1 [Davies Liu] address comments from @shivaram f61de71 [Davies Liu] Update pairRDD.R 3ef7cf3 [Davies Liu] use + instead of function(a,b) a+b 2f10a77 [Davies Liu] address comments from @cafreeman 9c2a062 [Davies Liu] mention R api together with Python API 23f751a [Davies Liu] Fill in SparkR examples in programming guide

@shivaram

…uide sqlCtx -> sqlContext You can check the docs by: ``` $ cd docs $ SKIP_SCALADOC=1 jekyll serve ``` cc shivaram Author: Davies Liu <[email protected]> Closes apache#5442 from davies/r_docs and squashes the following commits: 7a12ec6 [Davies Liu] remove rdd in R docs 8496b26 [Davies Liu] remove the docs related to RDD e23b9d6 [Davies Liu] delete R docs for RDD API 222e4ff [Davies Liu] Merge branch 'master' into r_docs 89684ce [Davies Liu] Merge branch 'r_docs' of github.com:davies/spark into r_docs f0a10e1 [Davies Liu] address comments from @shivaram f61de71 [Davies Liu] Update pairRDD.R 3ef7cf3 [Davies Liu] use + instead of function(a,b) a+b 2f10a77 [Davies Liu] address comments from @cafreeman 9c2a062 [Davies Liu] mention R api together with Python API 23f751a [Davies Liu] Fill in SparkR examples in programming guide

@shivaram

…uide sqlCtx -> sqlContext You can check the docs by: ``` $ cd docs $ SKIP_SCALADOC=1 jekyll serve ``` cc shivaram Author: Davies Liu <[email protected]> Closes apache#5442 from davies/r_docs and squashes the following commits: 7a12ec6 [Davies Liu] remove rdd in R docs 8496b26 [Davies Liu] remove the docs related to RDD e23b9d6 [Davies Liu] delete R docs for RDD API 222e4ff [Davies Liu] Merge branch 'master' into r_docs 89684ce [Davies Liu] Merge branch 'r_docs' of github.com:davies/spark into r_docs f0a10e1 [Davies Liu] address comments from @shivaram f61de71 [Davies Liu] Update pairRDD.R 3ef7cf3 [Davies Liu] use + instead of function(a,b) a+b 2f10a77 [Davies Liu] address comments from @cafreeman 9c2a062 [Davies Liu] mention R api together with Python API 23f751a [Davies Liu] Fill in SparkR examples in programming guide

Fill in SparkR examples in programming guide

23f751a

sqlCtx -> sqlContext

mention R api together with Python API

9c2a062

cafreeman reviewed Apr 10, 2015
View reviewed changes

shivaram reviewed Apr 10, 2015
View reviewed changes

shivaram mentioned this pull request Apr 10, 2015

[SparkR-239] buildSchema and field functions amplab-extras/SparkR-pkg#235

Merged

address comments from @cafreeman

2f10a77

use + instead of function(a,b) a+b

3ef7cf3

shivaram reviewed Apr 10, 2015
View reviewed changes

Update pairRDD.R

f61de71

shivaram reviewed May 18, 2015
View reviewed changes

remove the docs related to RDD

8496b26

remove rdd in R docs

7a12ec6

shivaram reviewed May 23, 2015
View reviewed changes

asfgit closed this in 7af3818 May 23, 2015

		@@ -54,6 +54,15 @@ Example applications are also provided in Python. For example,

		./bin/spark-submit examples/src/main/python/pi.py 10

		Spark also provides a R API. To run Spark interactively in a R interpreter, use

		@@ -491,6 +573,37 @@ for teenName in teenNames.collect():

		</div>

		<div data-lang="r" markdown="1">


		{% highlight r %}

		df <- laodDF(sqlContext, source="jdbc", url="jdbc:postgresql:dbserver", dbtable="schema.tablename")

[SPARK-6806] [SparkR] [Docs] Fill in SparkR examples in programming guide #5442

[SPARK-6806] [SparkR] [Docs] Fill in SparkR examples in programming guide #5442

Uh oh!

Conversation

davies commented Apr 9, 2015

Uh oh!

SparkQA commented Apr 9, 2015

Uh oh!

SparkQA commented Apr 9, 2015

Uh oh!

shivaram commented Apr 9, 2015

Uh oh!

SparkQA commented Apr 9, 2015

Uh oh!

AmplabJenkins commented Apr 9, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

AmplabJenkins commented Apr 10, 2015

Uh oh!

cafreeman Apr 10, 2015

Choose a reason for hiding this comment

Uh oh!

cafreeman commented Apr 10, 2015

Uh oh!

shivaram Apr 10, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

SparkQA commented Apr 10, 2015

Uh oh!

AmplabJenkins commented Apr 10, 2015

Uh oh!

shivaram Apr 10, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

shivaram May 18, 2015

Choose a reason for hiding this comment

Uh oh!

davies May 19, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015