[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size #1025

dorx · 2014-06-09T21:06:47Z

Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement.

Reviewer comments addressed: - commons-math3 is now a test-only dependency. bumped up to v3.3 - comments added to explain what computeFraction is doing - fixed the unit for computeFraction to use BinomialDitro for without replacement sampling - stylistic fixes

…exact sample size

AmplabJenkins · 2014-06-09T21:07:50Z

Build triggered.

AmplabJenkins · 2014-06-09T21:17:51Z

Build triggered.

AmplabJenkins · 2014-06-09T21:32:49Z

Build triggered.

rxin · 2014-06-09T21:36:59Z

core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala

@@ -46,7 +50,8 @@ import org.apache.spark.Partitioner.defaultPartitioner
 import org.apache.spark.SparkContext._
 import org.apache.spark.partial.{BoundedDouble, PartialResult}
 import org.apache.spark.serializer.Serializer
-import org.apache.spark.util.SerializableHyperLogLog
+import org.apache.spark.util.{Utils, SerializableHyperLogLog}


SerializableHyperLogLog no longer exists

AmplabJenkins · 2014-06-09T22:37:50Z

Build started.

AmplabJenkins · 2014-06-09T22:39:35Z

Build finished.

AmplabJenkins · 2014-06-09T22:39:35Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15579/

AmplabJenkins · 2014-06-09T22:39:36Z

Build started.

AmplabJenkins · 2014-06-09T22:41:13Z

Build finished.

AmplabJenkins · 2014-06-09T22:41:13Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15581/

AmplabJenkins · 2014-06-09T22:41:14Z

Build started.

AmplabJenkins · 2014-06-09T23:50:23Z

Build finished.

AmplabJenkins · 2014-06-09T23:50:24Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15582/

…to the aggregate function

AmplabJenkins · 2014-06-10T00:42:49Z

Build triggered.

AmplabJenkins · 2014-06-10T02:51:14Z

Build started.

AmplabJenkins · 2014-06-10T04:06:30Z

Build finished. All automated tests passed.

AmplabJenkins · 2014-06-10T04:06:30Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15589/

mengxr · 2014-06-10T05:27:12Z

core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala

+
+import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}
+
+private[spark] object PoissonBounds {


Though private, this object needs some documentation. I understand this is a little unfair ... :p

mateiz · 2014-07-25T05:22:30Z

Well, the other "sample" functions are already approximate anyway. I kind of like this here because it conveys that it's more expensive. The other thing is that if we want the Exact one to be experimental, we can't just make it a parameter.

fix poisson mean for waitlisting add unit tests for Java

mengxr · 2014-07-27T23:48:34Z

@dorx I removed commons-math3 from dependencies, separated sampleByKey and sampleByKeyExact, and corrected the math in waitlisting in sampling with replacement.

Other than those changes, the refactoring was completely based on your implementation. I did some renaming but I'm not taking any credit for the implementation. Some of the refactoring was unnecessary, which was partially because I did this around 5am and my head was not clear. I'm really sorry for the unnecessary code changes.

Since this is apparently your contribution, do you mind re-opening the PR? Thanks!

SparkQA · 2014-07-28T18:38:53Z

QA tests have started for PR 1025. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull

SparkQA · 2014-07-28T18:44:15Z

QA tests have started for PR 1025. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull

SparkQA · 2014-07-28T18:52:24Z

QA results for PR 1025:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17296/consoleFull

SparkQA · 2014-07-28T19:08:46Z

QA tests have started for PR 1025. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull

SparkQA · 2014-07-28T19:45:57Z

QA results for PR 1025:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17300/consoleFull

SparkQA · 2014-07-28T20:21:38Z

QA results for PR 1025:
- This patch FAILED unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17293/consoleFull

SparkQA · 2014-07-28T22:13:48Z

QA tests have started for PR 1025. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull

SparkQA · 2014-07-28T23:02:08Z

QA results for PR 1025:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17307/consoleFull

mengxr · 2014-07-29T16:25:24Z

core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala

+   */
+  def getLowerBound(delta: Double, n: Long, fraction: Double): Double = {
+    val gamma = - math.log(delta) / n * (2.0 / 3.0)
+    math.max(minSamplingRate,


It should be okay to return 0 here. The minSamplingRate should be applied to getUpperBound.

SparkQA · 2014-07-29T18:53:54Z

QA tests have started for PR 1025. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull

SparkQA · 2014-07-29T19:42:08Z

QA results for PR 1025:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17368/consoleFull

mengxr · 2014-07-29T19:50:41Z

LGTM. Merged into master. Thanks!!

…exact sample size Implemented stratified sampling that guarantees exact sample size using ScaRSR with two passes over the RDD for sampling without replacement and three passes for sampling with replacement. Author: Doris Xin <[email protected]> Author: Xiangrui Meng <[email protected]> Closes apache#1025 from dorx/stratified and squashes the following commits: 245439e [Doris Xin] moved minSamplingRate to getUpperBound eaf5771 [Doris Xin] bug fixes. 17a381b [Doris Xin] fixed a merge issue and a failed unit ea7d27f [Doris Xin] merge master b223529 [Xiangrui Meng] use approx bounds for poisson fix poisson mean for waitlisting add unit tests for Java b3013a4 [Xiangrui Meng] move math3 back to test scope eecee5f [Doris Xin] Merge branch 'master' into stratified f4c21f3 [Doris Xin] Reviewer comments a10e68d [Doris Xin] style fix a2bf756 [Doris Xin] Merge branch 'master' into stratified 680b677 [Doris Xin] use mapPartitionWithIndex instead 9884a9f [Doris Xin] style fix bbfb8c9 [Doris Xin] Merge branch 'master' into stratified ee9d260 [Doris Xin] addressed reviewer comments 6b5b10b [Doris Xin] Merge branch 'master' into stratified 254e03c [Doris Xin] minor fixes and Java API. 4ad516b [Doris Xin] remove unused imports from PairRDDFunctions bd9dc6e [Doris Xin] unit bug and style violation fixed 1fe1cff [Doris Xin] Changed fractionByKey to a map to enable arg check 944a10c [Doris Xin] [SPARK-2145] Add lower bound on sampling rate 0214a76 [Doris Xin] cleanUp 90d94c0 [Doris Xin] merge master 9e74ab5 [Doris Xin] Separated out most of the logic in sampleByKey 7327611 [Doris Xin] merge master 50581fc [Doris Xin] added a TODO for logging in python 46f6c8c [Doris Xin] fixed the NPE caused by closures being cleaned before being passed into the aggregate function 7e1a481 [Doris Xin] changed the permission on SamplingUtil 1d413ce [Doris Xin] fixed checkstyle issues 9ee94ee [Doris Xin] [SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size e3fd6a6 [Doris Xin] Merge branch 'master' into takeSample 7cab53a [Doris Xin] fixed import bug in rdd.py ffea61a [Doris Xin] SPARK-1939: Refactor takeSample method in RDD 1441977 [Doris Xin] SPARK-1939 Refactor takeSample method in RDD to use ScaSRS

…ommand (#1025) * check auth for subquerys in condition * add skip auth method for dataframe * add skip auth method for dataframe * add skip auth method for dataframe * add skip auth method for dataframe

…#1025) Co-authored-by: Egor Krivokon <>

dorx added 5 commits May 29, 2014 15:22

SPARK-1939 Refactor takeSample method in RDD to use ScaSRS

1441977

fixed import bug in rdd.py

7cab53a

Merge branch 'master' into takeSample

e3fd6a6

[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees …

9ee94ee

…exact sample size

fixed checkstyle issues

1d413ce

changed the permission on SamplingUtil

7e1a481

rxin reviewed Jun 9, 2014
View reviewed changes

fixed the NPE caused by closures being cleaned before being passed in…

46f6c8c

…to the aggregate function

mengxr reviewed Jun 10, 2014
View reviewed changes

use approx bounds for poisson

b223529

fix poisson mean for waitlisting add unit tests for Java

dorx closed this Jul 26, 2014

dorx reopened this Jul 28, 2014

merge master

ea7d27f

fixed a merge issue and a failed unit

17a381b

bug fixes.

eaf5771

mengxr reviewed Jul 29, 2014
View reviewed changes

moved minSamplingRate to getUpperBound

245439e

asfgit closed this in dc96536 Jul 29, 2014

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

[SPARK-1079] MaprFs jar is present in Spark-3.2.0/Spark-3.3.0 (apache…

c2af7da

…#1025) Co-authored-by: Egor Krivokon <>

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

[SPARK-1079] MaprFs jar is present in Spark-3.2.0/Spark-3.3.0 (apache…

0f2908b

…#1025) Co-authored-by: Egor Krivokon <>


		import org.apache.commons.math3.distribution.{PoissonDistribution, NormalDistribution}

		private[spark] object PoissonBounds {

[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size #1025

[SPARK-2082] stratified sampling in PairRDDFunctions that guarantees exact sample size #1025

Uh oh!

Conversation

dorx commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

rxin Jun 9, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

mengxr Jun 10, 2014

Choose a reason for hiding this comment

Uh oh!

mateiz commented Jul 25, 2014

Uh oh!

mengxr commented Jul 27, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

SparkQA commented Jul 28, 2014

Uh oh!

mengxr Jul 29, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

mengxr commented Jul 29, 2014

Uh oh!

Uh oh!