Skip to content

[SPARK-4233] [SQL] UDAF Interface Refactoring #5542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 20 commits into from
+1,685 −4
Closed

[SPARK-4233] [SQL] UDAF Interface Refactoring #5542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
440b689
migrate to support both version of UDAF
chenghao-intel Apr 11, 2015
7fb0662
Update the unit test to comment out the not support ones
chenghao-intel Apr 11, 2015
f118ffc
update the interface name
chenghao-intel Apr 13, 2015
f0b9ec0
change the update method from Any to Row
chenghao-intel Apr 15, 2015
bee0f95
move the distinct into the udaf
chenghao-intel Apr 15, 2015
760164e
simpify the aggregate expression by uing the Projection
chenghao-intel Apr 16, 2015
0849ca3
revert the uncessary changes
chenghao-intel Apr 16, 2015
472a440
Add Unit test
chenghao-intel Apr 16, 2015
241aee1
Add some doc
chenghao-intel Apr 16, 2015
de96a13
style issues
chenghao-intel Apr 16, 2015
483b381
more style issues
chenghao-intel Apr 16, 2015
58b1481
fix bug in the for unit test
chenghao-intel Apr 16, 2015
39a6243
use BufferAndKey class manully maitain the MutableRow
chenghao-intel Apr 17, 2015
5b01518
fix bug of with BufferAndKeys
chenghao-intel Apr 17, 2015
feac4d0
Add golden files
chenghao-intel Apr 17, 2015
ec7deaa
enable more unit test
chenghao-intel Apr 21, 2015
393b0d1
disable the codegen for aggregate2 in unit test
chenghao-intel Apr 21, 2015
021431f
Add more unit test
chenghao-intel Apr 24, 2015
8ad5fc5
rebase to the latest master
chenghao-intel May 18, 2015
68dd625
rebase again
chenghao-intel Jun 8, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,16 @@ public Row copy() {
}

@Override
public MutableRow makeMutable() {
GenericMutableRow mr = new GenericMutableRow(this.size());
for (int i = 0; i < mr.size(); ++i) {
mr.update(i, get(i));
}

return mr;
}

@Override
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation on this line looks off

public boolean anyNull() {
return BitSetMethods.anySet(baseObject, baseOffset, bitSetWidthInBytes);
}
Expand Down
13 changes: 12 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ package org.apache.spark.sql

import scala.util.hashing.MurmurHash3

import org.apache.spark.sql.catalyst.expressions.GenericRow
import org.apache.spark.sql.catalyst.expressions.{GenericMutableRow, MutableRow, GenericRow}
import org.apache.spark.sql.types.StructType

object Row {
Expand Down Expand Up @@ -348,6 +348,17 @@ trait Row extends Serializable {
*/
def copy(): Row

def makeMutable(): MutableRow = {
val totalSize = length
val copiedValues = new Array[Any](totalSize)
var i = 0
while(i < totalSize) {
copiedValues(i) = apply(i)
i += 1
}
new GenericMutableRow(copiedValues)
}

/** Returns true if there are any NULL values in this row. */
def anyNull: Boolean = {
val len = length
Expand Down
Loading