SPARK-6548 Adding stddev to DataFrame functions #6297

JihongMA · 2015-05-20T21:50:11Z

Adding STDDEV support for DataFrame using 1-pass online /parallel algorithm to compute variance. Please review the code change.

This reverts commit c40701a.

This reverts commit 3e7d889.

This reverts commit 634d6a7.

This reverts commit 125f3ae.

This reverts commit dfaa971.

This reverts commit ace454d.

This reverts commit 9c84695. Conflicts: docs/running-on-yarn.md

This reverts commit a399aa6. Conflicts: docs/running-on-yarn.md

marmbrus · 2015-06-23T01:06:46Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala

@@ -292,6 +293,7 @@ class SqlParser extends AbstractSparkSQLParser with DataTypeParser {
    | AVG   ~ "(" ~> expression <~ ")" ^^ { case exp => Average(exp) }
    | MIN   ~ "(" ~> expression <~ ")" ^^ { case exp => Min(exp) }
    | MAX   ~ "(" ~> expression <~ ")" ^^ { case exp => Max(exp) }
+    | STDDEV  ~ "(" ~> expression <~ ")" ^^ { case exp => Stddev(exp)}


We have changed how these plug in. You'll need to change the FunctionRegistry now.

JihongMA · 2015-07-24T20:56:18Z

Please don't test it yet, need to make change to accomodate API change introduced by other JIRA.

SparkQA · 2015-07-24T21:07:19Z

Test build #38399 has finished for PR 6297 at commit 87fd2dc.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends Serializable
- case class Stddev(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
- case class ComputePartialStd(child: Expression) extends AggregateExpression
- case class CombinePartialStd(child: Expression) extends AggregateExpression
- case class ComputePartialStdFunction (
- case class CombinePartialStdFunction(
- case class StddevFunction(
- class GenericRow(protected[sql] val values: Array[Any]) extends Row
- class GenericInternalRow(protected[sql] val values: Array[Any]) extends InternalRow
- class GenericInternalRowWithSchema(values: Array[Any], val schema: StructType)
- class GenericMutableRow(val values: Array[Any]) extends MutableRow

yhuai · 2015-07-29T14:55:04Z

@JihongMA Will you get time to implement the function based on the new API? It will be good if we can merge it before the 1.5 deadline for new features (end of this month).

SparkQA · 2015-08-28T06:45:58Z

Test build #41730 has finished for PR 6297 at commit 25425ac.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(child: Expression, isSample: Boolean) extends UnaryExpression with AggregateExpression1
- case class MergePartialStdFunction(
- case class StddevFunction(

SparkQA · 2015-08-28T08:59:13Z

Test build #41732 has finished for PR 6297 at commit f4c725c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(

SparkQA · 2015-08-28T18:20:40Z

Test build #41748 has finished for PR 6297 at commit 0902ceb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(

SparkQA · 2015-09-04T17:17:34Z

Test build #42006 has finished for PR 6297 at commit a81d0fc.

This patch fails R style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(

JihongMA · 2015-09-04T17:42:23Z

R style check failure is caused by commit of SPARK-8951

SparkQA · 2015-09-06T03:08:42Z

Test build #42062 has finished for PR 6297 at commit 6035648.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(

davies · 2015-09-10T23:35:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala

+  override def inputTypes: Seq[AbstractDataType] = Seq(TypeCollection(NumericType, NullType))
+
+  private val resultType = child.dataType match {
+    case DecimalType.Fixed(p, s) =>


I think it should always return Double, because Sqrt() only works with Double, also other databases just return Double/float.

SparkQA · 2015-09-12T06:54:24Z

Test build #42366 has finished for PR 6297 at commit 6351fc8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Stddev(child: Expression) extends StddevAgg(child)
- case class StddevPop(child: Expression) extends StddevAgg(child)
- case class StddevSamp(child: Expression) extends StddevAgg(child)
- abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
- abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
- case class Stddev(child: Expression) extends StddevAgg1(child)
- case class StddevPop(child: Expression) extends StddevAgg1(child)
- case class StddevSamp(child: Expression) extends StddevAgg1(child)
- case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
- case class ComputePartialStdFunction (
- case class MergePartialStd(
- case class MergePartialStdFunction(
- case class StddevFunction(

davies · 2015-09-12T17:16:41Z

LGTM, merging this into master, thanks!

JihongMA and others added 29 commits May 5, 2015 21:17

SPARK-7265 Improving documentation for Spark SQL Hive support

a399aa6

SPARK-7265 address review comment

9c84695

Merge remote-tracking branch 'upstream/master'

7b842e6

Merge remote-tracking branch 'upstream/master'

d5bf3f5

Merge remote-tracking branch 'upstream/master'

9ea0832

SPARK-7265 take out PySpark on YARN limitation

ace454d

SPARK-7265 minor fix of the content

dfaa971

Merge remote-tracking branch 'upstream/master'

a38e9f2

SPARK-6548 add stddev for datafram api

125f3ae

SPARK-6548 Add stddev for DataFrame

634d6a7

Merge remote-tracking branch 'upstream/master'

213c60e

SPARK-6548 minor fix

3e7d889

SPARK-6548 minor fix

c40701a

Merge remote-tracking branch 'upstream/master'

9d9bdcf

Merge remote branch 'upstream/master'

7adc7a3

Revert "SPARK-6548 minor fix"

c712568

This reverts commit c40701a.

Revert "SPARK-6548 minor fix"

665ce35

This reverts commit 3e7d889.

Revert "SPARK-6548 Add stddev for DataFrame"

319bd23

This reverts commit 634d6a7.

Revert "SPARK-6548 add stddev for datafram api"

f784358

This reverts commit 125f3ae.

Revert "SPARK-7265 minor fix of the content"

e7d2c0d

This reverts commit dfaa971.

Revert "SPARK-7265 take out PySpark on YARN limitation"

2ed3921

This reverts commit ace454d.

Revert "SPARK-7265 address review comment"

430bcfc

This reverts commit 9c84695. Conflicts: docs/running-on-yarn.md

Revert "SPARK-7265 Improving documentation for Spark SQL Hive support"

3790855

This reverts commit a399aa6. Conflicts: docs/running-on-yarn.md

Merge remote branch 'upstream/master'

e11ea50

fix running-on-yarn.md

1c871ff

merging runnin-on-yarn.md

3ee3c03

Merge remote branch 'upstream/master'

8204410

SPARK-6548 Add Stddev support as Catalyst expression in DataFrame

a73f041

Merge remote-tracking branch 'upstream/master'

df34292

marmbrus reviewed Jun 23, 2015
View reviewed changes

JihongMA and others added 10 commits August 3, 2015 18:30

merge spark upstream

4f3f4d3

stddev_pop & stddev_samp support

37f7366

rebase with upstream

849bf4e

minor code change to accomodate rebase

ce9226a

rebase with trunk

1f834e5

Merge remote-tracking branch 'upstream/master'

f343664

Merge remote-tracking branch 'upstream/master'

de18ed7

adding aggregatefunction2 support for stddev

abe5da1

rebase with trunk

3f78459

Merge branch 'SPARK-SQL-NEW', remote branch 'origin' into SPARK-SQL-NEW

25425ac

minor code style fix

f4c725c

update tests

0902ceb

string promotion & null value handling

a81d0fc

rebase with upstream

6035648

davies reviewed Sep 10, 2015
View reviewed changes

address Davides's comment

6351fc8

asfgit closed this in f4a2280 Sep 12, 2015

davies mentioned this pull request Sep 29, 2015

[SPARK-9673][SQL] Sample standard deviation aggregation function #8058

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SPARK-6548 Adding stddev to DataFrame functions #6297

SPARK-6548 Adding stddev to DataFrame functions #6297

Uh oh!

JihongMA commented May 20, 2015

Uh oh!

marmbrus Jun 23, 2015

Uh oh!

JihongMA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

yhuai commented Jul 29, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Sep 4, 2015

Uh oh!

JihongMA commented Sep 4, 2015

Uh oh!

SparkQA commented Sep 6, 2015

Uh oh!

davies Sep 10, 2015

Uh oh!

SparkQA commented Sep 12, 2015

Uh oh!

davies commented Sep 12, 2015

Uh oh!

Uh oh!

SPARK-6548 Adding stddev to DataFrame functions #6297

SPARK-6548 Adding stddev to DataFrame functions #6297

Uh oh!

Conversation

JihongMA commented May 20, 2015

Uh oh!

marmbrus Jun 23, 2015

Choose a reason for hiding this comment

Uh oh!

JihongMA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

yhuai commented Jul 29, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Aug 28, 2015

Uh oh!

SparkQA commented Sep 4, 2015

Uh oh!

JihongMA commented Sep 4, 2015

Uh oh!

SparkQA commented Sep 6, 2015

Uh oh!

davies Sep 10, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 12, 2015

Uh oh!

davies commented Sep 12, 2015

Uh oh!

Uh oh!