[SPARK-2650][SQL] Build column buffers in smaller batches #1880

marmbrus · 2014-08-10T21:22:03Z

No description provided.

SparkQA · 2014-08-10T21:24:39Z

QA tests have started for PR 1880. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18285/consoleFull

SparkQA · 2014-08-10T21:27:52Z

QA results for PR 1880:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18285/consoleFull

SparkQA · 2014-08-10T21:34:41Z

QA tests have started for PR 1880. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18286/consoleFull

SparkQA · 2014-08-10T22:49:40Z

QA results for PR 1880:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18286/consoleFull

SparkQA · 2014-08-10T22:54:43Z

QA tests have started for PR 1880. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18288/consoleFull

SparkQA · 2014-08-11T00:08:11Z

QA results for PR 1880:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18288/consoleFull

liancheng · 2014-08-11T01:26:33Z

sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala

+      // Find the ordinals of the requested columns.  If none are requested, use the first.
+      val requestedColumns =
+        if (attributes.isEmpty) {
+          Seq(0)


Maybe we can use the narrowest one instead of the 1st one by checking default sizes of columns:

val narrowest = relation.output.indices.minBy { i => ColumnType(relation.output(i).dataType).defaultSize } Seq(narrowest)

Yeah, that would be better. Really though I think we should use statistics from #1883 to skip decoding entirely.

liancheng · 2014-08-11T02:55:54Z

I believe this PR can alleviate OOMs a lot. Below are some ideas to make in-memory columnar store more memory efficient, and can be done in separate PRs based on this one.

While building column buffers in batch, we still uses 1MB as initial column buffer size for each column (defined as ColumnBuilder.DEFAULT_INITIAL_BUFFER_SIZE). Say T tasks are running in parallel to squeeze a table with C columns into memory, we allocate at least T * C * 1MB for each batch.

The initial column buffer size estimation used in Shark can be useful, but unfortunately the implementation is actually buggy, and usually gives fairly small initial buffer size. A more reasonable estimation heuristics could be:

Let D[i] be the default size of the i-th column
Let I = sum(D[i]) * batchSize
Default column buffer size for the i-th column is S[i] = I * D[i] / sum(D[i])

This estimation is precise for all primitive types whose default sizes equals to their actual sizes since the row number (i.e. batchSize) in a batch is known.

liancheng · 2014-08-11T03:00:08Z

Ah, just realized I made things too complex... Just use columnType.defaultSize * batchSize as the initial column buffer size, it's equivalent to the verbose version above.

liancheng · 2014-08-11T03:02:14Z

sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala

+      new Iterator[Array[ByteBuffer]] {
+        def next() = {
+          val columnBuilders = output.map { attribute =>
+            ColumnBuilder(ColumnType(attribute.dataType).typeId, 0, attribute.name, useCompression)


A more precise initial buffer size can be used here:

val columnType = ColumnType(attribute.dataType) ColumnBuilder(columnType.typeId, columnType.defaultSize * batchSize, attribute.name, useCompression)

marmbrus · 2014-08-12T03:20:39Z

@liancheng, thanks for reviewing! Would you mind creating a JIRA/followup PR to set the defaults correctly as you propose?

marmbrus · 2014-08-12T03:22:19Z

Merged to master and 1.1

Author: Michael Armbrust <[email protected]> Closes #1880 from marmbrus/columnBatches and squashes the following commits: 0649987 [Michael Armbrust] add test 4756fad [Michael Armbrust] fix compilation 2314532 [Michael Armbrust] Build column buffers in smaller batches (cherry picked from commit bad21ed) Signed-off-by: Michael Armbrust <[email protected]>

liancheng · 2014-08-12T06:11:29Z

Opened #1901 for precise initial buffer size estimation.

…memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian <[email protected]> Closes #1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer (cherry picked from commit 376a82e) Signed-off-by: Michael Armbrust <[email protected]>

…memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian <[email protected]> Closes #1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer

@transient

…tions This PR is based on #1883 authored by marmbrus. Key differences: 1. Batch pruning instead of partition pruning When #1883 was authored, batched column buffer building (#1880) hadn't been introduced. This PR combines these two and provide partition batch level pruning, which leads to smaller memory footprints and can generally skip more elements. The cost is that the pruning predicates are evaluated more frequently (partition number multiplies batch number per partition). 1. More filters are supported Filter predicates consist of `=`, `<`, `<=`, `>`, `>=` and their conjunctions and disjunctions are supported. Author: Cheng Lian <[email protected]> Closes #2188 from liancheng/in-mem-batch-pruning and squashes the following commits: 68cf019 [Cheng Lian] Marked sqlContext as @transient 4254f6c [Cheng Lian] Enables in-memory partition pruning in PartitionBatchPruningSuite 3784105 [Cheng Lian] Overrides InMemoryColumnarTableScan.sqlContext d2a1d66 [Cheng Lian] Disables in-memory partition pruning by default 062c315 [Cheng Lian] HiveCompatibilitySuite code cleanup 16b77bf [Cheng Lian] Fixed pruning predication conjunctions and disjunctions 16195c5 [Cheng Lian] Enabled both disjunction and conjunction 89950d0 [Cheng Lian] Worked around Scala style check 9c167f6 [Cheng Lian] Minor code cleanup 3c4d5c7 [Cheng Lian] Minor code cleanup ea59ee5 [Cheng Lian] Renamed PartitionSkippingSuite to PartitionBatchPruningSuite fc517d0 [Cheng Lian] More test cases 1868c18 [Cheng Lian] Code cleanup, bugfix, and adding tests cb76da4 [Cheng Lian] Added more predicate filters, fixed table scan stats for testing purposes 385474a [Cheng Lian] Merge branch 'inMemStats' into in-mem-batch-pruning

Author: Michael Armbrust <[email protected]> Closes apache#1880 from marmbrus/columnBatches and squashes the following commits: 0649987 [Michael Armbrust] add test 4756fad [Michael Armbrust] fix compilation 2314532 [Michael Armbrust] Build column buffers in smaller batches

…memory column buffer This is a follow up of apache#1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian <[email protected]> Closes apache#1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer

@transient

…tions This PR is based on apache#1883 authored by marmbrus. Key differences: 1. Batch pruning instead of partition pruning When apache#1883 was authored, batched column buffer building (apache#1880) hadn't been introduced. This PR combines these two and provide partition batch level pruning, which leads to smaller memory footprints and can generally skip more elements. The cost is that the pruning predicates are evaluated more frequently (partition number multiplies batch number per partition). 1. More filters are supported Filter predicates consist of `=`, `<`, `<=`, `>`, `>=` and their conjunctions and disjunctions are supported. Author: Cheng Lian <[email protected]> Closes apache#2188 from liancheng/in-mem-batch-pruning and squashes the following commits: 68cf019 [Cheng Lian] Marked sqlContext as @transient 4254f6c [Cheng Lian] Enables in-memory partition pruning in PartitionBatchPruningSuite 3784105 [Cheng Lian] Overrides InMemoryColumnarTableScan.sqlContext d2a1d66 [Cheng Lian] Disables in-memory partition pruning by default 062c315 [Cheng Lian] HiveCompatibilitySuite code cleanup 16b77bf [Cheng Lian] Fixed pruning predication conjunctions and disjunctions 16195c5 [Cheng Lian] Enabled both disjunction and conjunction 89950d0 [Cheng Lian] Worked around Scala style check 9c167f6 [Cheng Lian] Minor code cleanup 3c4d5c7 [Cheng Lian] Minor code cleanup ea59ee5 [Cheng Lian] Renamed PartitionSkippingSuite to PartitionBatchPruningSuite fc517d0 [Cheng Lian] More test cases 1868c18 [Cheng Lian] Code cleanup, bugfix, and adding tests cb76da4 [Cheng Lian] Added more predicate filters, fixed table scan stats for testing purposes 385474a [Cheng Lian] Merge branch 'inMemStats' into in-mem-batch-pruning

Build column buffers in smaller batches

2314532

marmbrus changed the title ~~[SPARK-2650] Build column buffers in smaller batches~~ [SPARK-2650][SQL] Build column buffers in smaller batches Aug 10, 2014

fix compilation

4756fad

add test

0649987

liancheng reviewed Aug 11, 2014
View reviewed changes

asfgit closed this in bad21ed Aug 12, 2014

liancheng mentioned this pull request Aug 12, 2014

[SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer #1901

Closed

marmbrus deleted the columnBatches branch August 27, 2014 20:44

liancheng mentioned this pull request Aug 29, 2014

[SPARK-2961][SQL] Use statistics to prune batches within cached partitions #2188

Closed

szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Feb 7, 2024

rdar://119344310 Bump Boson version to 0.3.23 (apache#1880)

4c50466

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-2650][SQL] Build column buffers in smaller batches #1880

[SPARK-2650][SQL] Build column buffers in smaller batches #1880

Uh oh!

marmbrus commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Uh oh!

marmbrus Aug 11, 2014

Uh oh!

liancheng commented Aug 11, 2014

Uh oh!

liancheng commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Uh oh!

marmbrus commented Aug 12, 2014

Uh oh!

marmbrus commented Aug 12, 2014

Uh oh!

liancheng commented Aug 12, 2014

Uh oh!

Uh oh!

[SPARK-2650][SQL] Build column buffers in smaller batches #1880

[SPARK-2650][SQL] Build column buffers in smaller batches #1880

Uh oh!

Conversation

marmbrus commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 10, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Aug 11, 2014

Uh oh!

liancheng commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Aug 12, 2014

Uh oh!

marmbrus commented Aug 12, 2014

Uh oh!

liancheng commented Aug 12, 2014

Uh oh!

Uh oh!