[Spark-7511][MLLIB] pyspark ml seed param should be random by default or 42 is quite funny but not very random #6139

holdenk · 2015-05-14T02:51:48Z

No description provided.

… default rather than having None is note None in the generated code

AmplabJenkins · 2015-05-14T02:52:10Z

Merged build triggered.

AmplabJenkins · 2015-05-14T02:52:17Z

Merged build started.

SparkQA · 2015-05-14T02:54:07Z

Test build #32673 has started for PR 6139 at commit 0e3797e.

SparkQA · 2015-05-14T02:55:44Z

Test build #32673 has finished for PR 6139 at commit 0e3797e.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-14T02:55:45Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-14T02:55:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32673/
Test FAILed.

JoshRosen · 2015-05-14T03:58:15Z

AmplabJenkins · 2015-05-14T07:02:11Z

Merged build triggered.

AmplabJenkins · 2015-05-14T07:02:18Z

Merged build started.

SparkQA · 2015-05-14T07:02:58Z

Test build #32688 has started for PR 6139 at commit 65eba21.

holdenk · 2015-05-14T07:22:09Z

@JoshRosen

:)

mengxr · 2015-05-14T08:47:08Z

python/pyspark/ml/feature.py

@@ -790,7 +790,7 @@ class Word2Vec(JavaEstimator, HasStepSize, HasMaxIter, HasSeed, HasInputCol, Has

    @keyword_only
    def __init__(self, vectorSize=100, minCount=5, numPartitions=1, stepSize=0.025, maxIter=1,
-                 seed=42, inputCol=None, outputCol=None):
+                 seed=None, inputCol=None, outputCol=None):
        """
        __init__(self, vectorSize=100, minCount=5, numPartitions=1, stepSize=0.025, maxIter=1,
                 seed=42, inputCol=None, outputCol=None)


We also need to update the docstring to keep the same signature.

Good point, I originally didn't because I though they might have been doctests (and I wanted to keep the tests predictable), but that doesn't seem to be the case. I'll update these :)

mengxr · 2015-05-14T08:48:14Z

@holdenk @JoshRosen Thanks for sharing!

SparkQA · 2015-05-14T08:49:00Z

Test build #32688 has finished for PR 6139 at commit 65eba21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-14T08:49:06Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-14T08:49:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32688/
Test PASSed.

mengxr · 2015-05-14T08:51:12Z

@holdenk Please add [MLLIB] to the PR title so it gets listed under "MLlib" on https://spark-prs.appspot.com/#mllib.

holdenk · 2015-05-14T21:00:57Z

@mengxr Thanks for revewing this! I'll update the PR title :)

jkbradley · 2015-05-14T21:04:40Z

Haha, I like this PR so far : )

AmplabJenkins · 2015-05-14T22:27:12Z

Merged build triggered.

AmplabJenkins · 2015-05-14T22:27:20Z

Merged build started.

AmplabJenkins · 2015-05-20T07:12:12Z

Merged build triggered.

AmplabJenkins · 2015-05-20T07:12:19Z

Merged build started.

SparkQA · 2015-05-20T07:12:43Z

Test build #33129 has started for PR 6139 at commit 591f8e5.

SparkQA · 2015-05-20T09:42:44Z

Test build #33129 timed out for PR 6139 at commit 591f8e5 after a configured wait of 150m.

AmplabJenkins · 2015-05-20T09:42:50Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T09:42:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33129/
Test FAILed.

mengxr · 2015-05-20T14:53:45Z

@tdas @davies There were timeouts in PySpark Streaming unit tests: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33129/consoleFull.

mengxr · 2015-05-20T14:53:51Z

test this please

AmplabJenkins · 2015-05-20T14:57:17Z

Merged build triggered.

AmplabJenkins · 2015-05-20T14:57:27Z

Merged build started.

SparkQA · 2015-05-20T14:58:12Z

Test build #33146 has started for PR 6139 at commit 591f8e5.

SparkQA · 2015-05-20T14:59:48Z

Test build #33146 has finished for PR 6139 at commit 591f8e5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-20T14:59:49Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T14:59:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33146/
Test FAILed.

holdenk · 2015-05-20T17:33:50Z

since we didn't change any Scala code failing the Scala style tests is unexpected. @AmplabJenkins retest this please.

AmplabJenkins · 2015-05-20T17:37:13Z

Merged build triggered.

AmplabJenkins · 2015-05-20T17:37:23Z

Merged build started.

SparkQA · 2015-05-20T17:38:06Z

Test build #33150 has started for PR 6139 at commit 591f8e5.

SparkQA · 2015-05-20T18:48:06Z

Test build #33150 has finished for PR 6139 at commit 591f8e5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-20T18:48:11Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-20T18:48:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33150/
Test FAILed.

SparkQA · 2015-05-20T20:23:56Z

Test build #837 has started for PR 6139 at commit 591f8e5.

SparkQA · 2015-05-20T22:09:08Z

Test build #837 has finished for PR 6139 at commit 591f8e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2015-05-20T22:15:34Z

Merging into master and branch-1.4 (branch-1.4 is merited since this changes algorithm behavior)
@holdenk Thanks!

…t or 42 is quite funny but not very random Author: Holden Karau <[email protected]> Closes #6139 from holdenk/SPARK-7511-pyspark-ml-seed-param-should-be-random-by-default-or-42-is-quite-funny-but-not-very-random and squashes the following commits: 591f8e5 [Holden Karau] specify old seed for doc tests 2470004 [Holden Karau] Fix a bunch of seeds with default values to have None as the default which will then result in using the hash of the class name cbad96d [Holden Karau] Add the setParams function that is used in the real code 423b8d7 [Holden Karau] Switch the test code to behave slightly more like production code. also don't check the param map value only check for key existence 140d25d [Holden Karau] remove extra space 926165a [Holden Karau] Add some missing newlines for pep8 style 8616751 [Holden Karau] merge in master 58532e6 [Holden Karau] its the __name__ method, also treat None values as not set 56ef24a [Holden Karau] fix test and regenerate base afdaa5c [Holden Karau] make sure different classes have different results 68eb528 [Holden Karau] switch default seed to hash of type of self 89c4611 [Holden Karau] Merge branch 'master' into SPARK-7511-pyspark-ml-seed-param-should-be-random-by-default-or-42-is-quite-funny-but-not-very-random 31cd96f [Holden Karau] specify the seed to randomforestregressor test e1b947f [Holden Karau] Style fixes ce90ec8 [Holden Karau] merge in master bcdf3c9 [Holden Karau] update docstring seeds to none and some other default seeds from 42 65eba21 [Holden Karau] pep8 fixes 0e3797e [Holden Karau] Make seed default to random in more places 213a543 [Holden Karau] Simplify the generated code to only include set default if there is a default rather than having None is note None in the generated code 1ff17c2 [Holden Karau] Make the seed random for HasSeed in python (cherry picked from commit 191ee47) Signed-off-by: Joseph K. Bradley <[email protected]>

…t or 42 is quite funny but not very random Author: Holden Karau <[email protected]> Closes apache#6139 from holdenk/SPARK-7511-pyspark-ml-seed-param-should-be-random-by-default-or-42-is-quite-funny-but-not-very-random and squashes the following commits: 591f8e5 [Holden Karau] specify old seed for doc tests 2470004 [Holden Karau] Fix a bunch of seeds with default values to have None as the default which will then result in using the hash of the class name cbad96d [Holden Karau] Add the setParams function that is used in the real code 423b8d7 [Holden Karau] Switch the test code to behave slightly more like production code. also don't check the param map value only check for key existence 140d25d [Holden Karau] remove extra space 926165a [Holden Karau] Add some missing newlines for pep8 style 8616751 [Holden Karau] merge in master 58532e6 [Holden Karau] its the __name__ method, also treat None values as not set 56ef24a [Holden Karau] fix test and regenerate base afdaa5c [Holden Karau] make sure different classes have different results 68eb528 [Holden Karau] switch default seed to hash of type of self 89c4611 [Holden Karau] Merge branch 'master' into SPARK-7511-pyspark-ml-seed-param-should-be-random-by-default-or-42-is-quite-funny-but-not-very-random 31cd96f [Holden Karau] specify the seed to randomforestregressor test e1b947f [Holden Karau] Style fixes ce90ec8 [Holden Karau] merge in master bcdf3c9 [Holden Karau] update docstring seeds to none and some other default seeds from 42 65eba21 [Holden Karau] pep8 fixes 0e3797e [Holden Karau] Make seed default to random in more places 213a543 [Holden Karau] Simplify the generated code to only include set default if there is a default rather than having None is note None in the generated code 1ff17c2 [Holden Karau] Make the seed random for HasSeed in python

holdenk added 3 commits May 13, 2015 18:56

Make the seed random for HasSeed in python

1ff17c2

Simplify the generated code to only include set default if there is a…

213a543

… default rather than having None is note None in the generated code

Make seed default to random in more places

0e3797e

pep8 fixes

65eba21

mengxr reviewed May 14, 2015
View reviewed changes

holdenk changed the title ~~[Spark-7511] pyspark ml seed param should be random by default or 42 is quite funny but not very random~~ [Spark-7511][MLLIB] pyspark ml seed param should be random by default or 42 is quite funny but not very random May 14, 2015

update docstring seeds to none and some other default seeds from 42

bcdf3c9

jkbradley mentioned this pull request May 20, 2015

[SPARK-7762] [MLLIB] set default value for outputCol #6289

Closed

asfgit closed this in 191ee47 May 20, 2015

[Spark-7511][MLLIB] pyspark ml seed param should be random by default or 42 is quite funny but not very random #6139

[Spark-7511][MLLIB] pyspark ml seed param should be random by default or 42 is quite funny but not very random #6139

Uh oh!

Conversation

holdenk commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

SparkQA commented May 14, 2015

Uh oh!

SparkQA commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

JoshRosen commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

SparkQA commented May 14, 2015

Uh oh!

holdenk commented May 14, 2015

Uh oh!

mengxr May 14, 2015

Choose a reason for hiding this comment

Uh oh!

holdenk May 14, 2015

Choose a reason for hiding this comment

Uh oh!

mengxr commented May 14, 2015

Uh oh!

SparkQA commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

mengxr commented May 14, 2015

Uh oh!

holdenk commented May 14, 2015

Uh oh!

jkbradley commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 14, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

mengxr commented May 20, 2015

Uh oh!

mengxr commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

SparkQA commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!

AmplabJenkins commented May 20, 2015

Uh oh!