[SPARK-7704] Updating Programming Guides per SPARK-4397 #6234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher)
|
Can one of the admins verify this patch? |
daisukebe
changed the title
|
OK to test |
|
Yes I believe that's correct. |
|
I remember we keep |
|
The docs are versioned though. I can understand that people using old Spark might still look at the latest docs, but then there are a number of problems of that form. I think/hope people would be careful to look at the right version if one version didn't seem to be working or making sense. |
| (the built-in tuples in the language, created by simply writing `(a, b)`), as long as you | ||
| import `org.apache.spark.SparkContext._` in your program to enable Spark's implicit | ||
| conversions. The key-value pair operations are available in the | ||
| (the built-in tuples in the language, created by simply writing `(a, b)`). The key-value pair operations are available in the | ||
| [PairRDDFunctions](api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions) class, | ||
| which automatically wraps around an RDD of tuples if you import the conversions. |
There was a problem hiding this comment.
if you import the conversions can be removed.
There was a problem hiding this comment.
(I can tap that in on merge, OK.)
|
Yes we should remove it. But maybe add a note somewhere to say "in Spark version xxx and below, you would need to explicitly ..." |
|
Thanks guys. Then, does adding the following make sense?
|
|
That looks good. |
| import org.apache.spark.SparkConf | ||
| {% endhighlight %} | ||
|
|
||
| (If the Spark version is prior to 1.3.0, user needs to explicitly import org.apache.spark.SparkContext._ to allow the implicit conversions.) |
There was a problem hiding this comment.
I suggest a slight rewording and putting the import in code font:
Before Spark 1.3.0, you need to explicitly import org.apache.spark.SparkContext._ to enable essential implicit conversions.
|
LGTM except the minor comment. But I think @srowen will handle it. |
asfgit
pushed a commit
that referenced
this pull request
May 19, 2015
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher) Author: Dice <[email protected]> Closes #6234 from daisukebe/patch-1 and squashes the following commits: b77ecd9 [Dice] fix a typo 45dfcd3 [Dice] rewording per Sean's advice a094bcf [Dice] Adding a note for users on any previous releases a29be5f [Dice] Updating Programming Guides per SPARK-4397 (cherry picked from commit 32fa611) Signed-off-by: Sean Owen <[email protected]>
jeanlyn
pushed a commit
to jeanlyn/spark
that referenced
this pull request
May 28, 2015
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher) Author: Dice <[email protected]> Closes apache#6234 from daisukebe/patch-1 and squashes the following commits: b77ecd9 [Dice] fix a typo 45dfcd3 [Dice] rewording per Sean's advice a094bcf [Dice] Adding a note for users on any previous releases a29be5f [Dice] Updating Programming Guides per SPARK-4397
jeanlyn
pushed a commit
to jeanlyn/spark
that referenced
this pull request
Jun 12, 2015
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher) Author: Dice <[email protected]> Closes apache#6234 from daisukebe/patch-1 and squashes the following commits: b77ecd9 [Dice] fix a typo 45dfcd3 [Dice] rewording per Sean's advice a094bcf [Dice] Adding a note for users on any previous releases a29be5f [Dice] Updating Programming Guides per SPARK-4397
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher) Author: Dice <[email protected]> Closes apache#6234 from daisukebe/patch-1 and squashes the following commits: b77ecd9 [Dice] fix a typo 45dfcd3 [Dice] rewording per Sean's advice a094bcf [Dice] Adding a note for users on any previous releases a29be5f [Dice] Updating Programming Guides per SPARK-4397
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The change per SPARK-4397 makes implicit objects in SparkContext to be found by the compiler automatically. So that we don't need to import the o.a.s.SparkContext._ explicitly any more and can remove some statements around the "implicit conversions" from the latest Programming Guides (1.3.0 and higher)