Skip to content

[SPARK-4348] [SPARK-4821] Backport PySpark random.py -> rand.py fix to branch-1.1 #4011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-4348] [SPARK-4821] Backport PySpark random.py -> rand.py fix to branch-1.1 #4011

wants to merge 2 commits into from

Conversation

JoshRosen
Copy link
Contributor

This backports #3216 and #3669 to branch-1.1 in order to fix the PySpark unit tests.

Davies Liu and others added 2 commits January 12, 2015 14:47
@JoshRosen
[SPARK-4348] [PySpark] [MLlib] rename random.py to rand.py
7ae5a1c 
This PR rename random.py to rand.py to avoid the side affects of conflict with random module, but still keep the same interface as before.

```
>>> from pyspark.mllib.random import RandomRDDs
```

```
$ pydoc pyspark.mllib.random
Help on module random in pyspark.mllib:
NAME
    random - Python package for random data generation.

FILE
    /Users/davies/work/spark/python/pyspark/mllib/rand.py

CLASSES
    __builtin__.object
        pyspark.mllib.random.RandomRDDs

    class RandomRDDs(__builtin__.object)
     |  Generator methods for creating RDDs comprised of i.i.d samples from
     |  some distribution.
     |
     |  Static methods defined here:
     |
     |  normalRDD(sc, size, numPartitions=None, seed=None)
```

cc mengxr

reference link: http://xion.org.pl/2012/05/06/hacking-python-imports/

Author: Davies Liu <[email protected]>

Closes apache#3216 from davies/random and squashes the following commits:

7ac4e8b [Davies Liu] rename random.py to rand.py

(cherry picked from commit ce0333f)
Signed-off-by: Josh Rosen <[email protected]>

Conflicts:
	python/pyspark/mllib/feature.py
	python/run-tests
@jkbradley @JoshRosen
[SPARK-4821] [mllib] [python] [docs] Fix for pyspark.mllib.rand doc
ace4cb6 
+ small doc edit
+ include edit to make IntelliJ happy

CC: davies  mengxr

Note to davies  -- this does not fix the "WARNING: Literal block expected; none found." warnings since that seems to involve spacing which IntelliJ does not like.  (Those warnings occur when generating the Python docs.)

Author: Joseph K. Bradley <[email protected]>

Closes apache#3669 from jkbradley/python-warnings and squashes the following commits:

4587868 [Joseph K. Bradley] fixed warning
8cb073c [Joseph K. Bradley] Updated based on davies recommendation
c51eca4 [Joseph K. Bradley] Updated rst file for pyspark.mllib.rand doc.  Small doc edit.  Small include edit to make IntelliJ happy.

Conflicts:
	python/docs/pyspark.streaming.rst
	python/pyspark/mllib/feature.py
@JoshRosen
Copy link
Contributor Author

/cc @davies @jkbradley

@SparkQA
Copy link

SparkQA commented Jan 12, 2015

QA tests have started for PR 4011 at commit ace4cb6.

  • This patch merges cleanly.

@davies
Copy link
Contributor

davies commented Jan 12, 2015

LGTM, thanks!

@andrewor14
Copy link
Contributor

Yay!

@SparkQA
Copy link

SparkQA commented Jan 13, 2015

QA tests have finished for PR 4011 at commit ace4cb6.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25432/
Test FAILed.

@JoshRosen
Copy link
Contributor Author

This fixed the PySpark unit tests, but the build is still failing due to a MiMa issue (which we hadn't noticed because PySpark failures were preventing those checks from running):

[error]  * method localAccums()scala.collection.mutable.Map in object org.apache.spark.Accumulators has now a different result type; was: scala.collection.mutable.Map, is now: java.lang.ThreadLocal
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.Accumulators.localAccums")
[info] spark-streaming-zeromq: found 0 potential binary incompatibilities
[info] spark-streaming-kafka: found 0 potential binary incompatibilities (filtered 3)
[info] spark-streaming-mqtt: found 0 potential binary incompatibilities
[info] spark-streaming-twitter: found 0 potential binary incompatibilities
[info] spark-streaming-flume: found 0 potential binary incompatibilities (filtered 2)
[info] spark-streaming: found 0 potential binary incompatibilities (filtered 5)
[info] spark-mllib: found 0 potential binary incompatibilities (filtered 45)
java.lang.RuntimeException: spark-core: Binary compatibility check failed!
    at scala.sys.package$.error(package.scala:27)
    at com.typesafe.tools.mima.plugin.SbtMima$.reportErrors(SbtMima.scala:64)
    at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23)
    at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23)
    at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:35)
    at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:34)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
    at sbt.std.Transform$$anon$4.work(System.scala:64)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.Execute.work(Execute.scala:244)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (core/*:mimaReportBinaryIssues) spark-core: Binary compatibility check failed!
[error] Total time: 43 s, completed Jan 12, 2015 4:11:07 PM

Fixing this should just be a matter of backporting an exclude from another branch, since the method in question was not public (this is due to how MiMa handles public fields in private[spark] classes).

I'm going to merge this now, then push a hotfix commit to address the MiMa issue.

@JoshRosen JoshRosen closed this Jan 13, 2015
@JoshRosen JoshRosen deleted the pyspark-rand-fix-1.1-backport branch January 13, 2015 00:16
asfgit pushed a commit that referenced this pull request Jan 13, 2015
@jkbradley @JoshRosen
[SPARK-4348] [SPARK-4821] Backport PySpark random.py -> rand.py fix t…
c98dc0e 
…o branch-1.1

This backports #3216 and #3669 to `branch-1.1` in order to fix the PySpark unit tests.

Author: Joseph K. Bradley <[email protected]>
Author: Davies Liu <[email protected]>

Closes #4011 from JoshRosen/pyspark-rand-fix-1.1-backport and squashes the following commits:

ace4cb6 [Joseph K. Bradley] [SPARK-4821] [mllib] [python] [docs] Fix for pyspark.mllib.rand doc
7ae5a1c [Davies Liu] [SPARK-4348] [PySpark] [MLlib] rename random.py to rand.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@JoshRosen @SparkQA @davies @andrewor14 @AmplabJenkins @jkbradley