Skip to content

[WIP] remove expensive api from InternalRow #6869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

davies
Copy link
Contributor

@davies davies commented Jun 18, 2015

This is a follow up PR for #6792, to use narrow down the APIs in InternalRow (remove those expensive ones).

@davies
Copy link
Contributor Author

davies commented Jun 18, 2015

cc @liancheng @rxin @JoshRosen

@cloud-fan
Copy link
Contributor

hi @davies , I met a bug at #6825 that we didn't handle BinaryType(byte[]) correctly at equals and hashCode. We should use java.util.Arrays.equals and java.util.Arrays.hashCode. Could you fix it here?

@SparkQA
Copy link

SparkQA commented Jun 18, 2015

Test build #35092 has finished for PR 6869 at commit 444382e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • trait Row extends InternalRow
    • abstract class InternalRow extends Serializable
    • final class SpecificMutableRow(val values: Array[MutableValue]) extends Row with MutableRow
    • class GenericRow(protected[sql] val values: Array[Any]) extends Row
    • abstract class InternalOutputWriter extends OutputWriter

}
abstract class InternalRow extends Serializable {
/** Number of elements in the Row. */
def size: Int = length
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need size? maybe just keep one of length or size.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree. It's internal and we don't need to follow scala convention to provide size.

@rxin
Copy link
Contributor

rxin commented Jun 18, 2015

@davies can you update the pr description to include what you actually removed?

@davies
Copy link
Contributor Author

davies commented Jun 18, 2015

@cloud-fan it will be fixed by #6876

* @throws ClassCastException when data type does not match.
* @throws NullPointerException when value is null.
*/
def getString(i: Int): String = getAs[UTF8String](i).toString
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it's internal row, should we just return UTF8String?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can only define getUTF8String at InternalRow and define getString at user-facing Row?

@davies
Copy link
Contributor Author

davies commented Jun 24, 2015

Close this, in favor of #7003

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants