Skip to content

SPARK-1917: fix PySpark import of scipy.special functions #866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

SPARK-1917: fix PySpark import of scipy.special functions #866

wants to merge 2 commits into from

Conversation

laserson
Copy link
Contributor

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@rxin
Copy link
Contributor

rxin commented May 23, 2014

@JoshRosen

@mateiz
Copy link
Contributor

mateiz commented May 28, 2014

@laserson can you add some tests for this? You can make them run only if SciPy is imported -- see the MLlib tests in mllib/tests.py for an example. But we need a way to verify that this is working.

@mateiz
Copy link
Contributor

mateiz commented May 28, 2014

Jenkins, this is ok to test

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15264/

@pwendell
Copy link
Contributor

Jenkins, retest this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@laserson
Copy link
Contributor Author

Will add tests tomorrow morning...

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15282/

@laserson
Copy link
Contributor Author

@mateiz test added

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15294/

@asfgit asfgit closed this in 5e98967 May 31, 2014
asfgit pushed a commit that referenced this pull request May 31, 2014
@laserson @mateiz
SPARK-1917: fix PySpark import of scipy.special functions
46825cd 
https://issues.apache.org/jira/browse/SPARK-1917

Author: Uri Laserson <[email protected]>

Closes #866 from laserson/SPARK-1917 and squashes the following commits:

d947e8c [Uri Laserson] Added test for scipy.special importing
1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special

(cherry picked from commit 5e98967)
Signed-off-by: Matei Zaharia <[email protected]>
@mateiz
Copy link
Contributor

mateiz commented May 31, 2014

Thanks Uri. Merged this into branch-1.0 as well as 0.9.

asfgit pushed a commit that referenced this pull request May 31, 2014
@laserson @mateiz
SPARK-1917: fix PySpark import of scipy.special functions
e03af41 
https://issues.apache.org/jira/browse/SPARK-1917

Author: Uri Laserson <[email protected]>

Closes #866 from laserson/SPARK-1917 and squashes the following commits:

d947e8c [Uri Laserson] Added test for scipy.special importing
1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special

Conflicts:
	python/pyspark/tests.py
markhamstra pushed a commit to markhamstra/spark that referenced this pull request Jun 6, 2014
@laserson @markhamstra
SPARK-1917: fix PySpark import of scipy.special functions
8e6ab49 
https://issues.apache.org/jira/browse/SPARK-1917

Author: Uri Laserson <[email protected]>

Closes apache#866 from laserson/SPARK-1917 and squashes the following commits:

d947e8c [Uri Laserson] Added test for scipy.special importing
1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special

Conflicts:
	python/pyspark/tests.py
@pwendell
Copy link
Contributor

@mateiz @laserson This patched introduced a change which breaks the unit tests on Python 2.6. The skipIf feature is not supported until Python 2.7

@laserson
Copy link
Contributor Author

Yes, that's annoying about python 2.6 Note that I just copied the same pattern that was present in spark/python/pyspark/mllib/tests.py. One option would be to import unittest2 if the interpreter is 2.6.

@pwendell
Copy link
Contributor

@laserson is it possible to just wrap the test in an "if statement"?

@laserson
Copy link
Contributor Author

I guess that would work, though I've never seen anyone do that. unittest2 was created to backport all of the new functionality in unittest to py2.6.

@mateiz
Copy link
Contributor

mateiz commented Jun 10, 2014

FYI we've also been using skipIf for a while in python/pyspark/mllib/tests.py. I guess you're trying Jenkins on 2.6 now?

@laserson
Copy link
Contributor Author

I created SPARK-2101 to address this.

@laserson
Copy link
Contributor Author

See #1042 for a patch.

pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@laserson @pdeyhim
SPARK-1917: fix PySpark import of scipy.special functions
a36d925 
https://issues.apache.org/jira/browse/SPARK-1917

Author: Uri Laserson <[email protected]>

Closes apache#866 from laserson/SPARK-1917 and squashes the following commits:

d947e8c [Uri Laserson] Added test for scipy.special importing
1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@laserson @conviva-zz
SPARK-1917: fix PySpark import of scipy.special functions
5b316b3 
https://issues.apache.org/jira/browse/SPARK-1917

Author: Uri Laserson <[email protected]>

Closes apache#866 from laserson/SPARK-1917 and squashes the following commits:

d947e8c [Uri Laserson] Added test for scipy.special importing
1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@laserson @AmplabJenkins @rxin @mateiz @pwendell