[SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution #10170

JoshRosen · 2015-12-07T04:10:17Z

This patch fixes a bug in the eviction of storage memory by execution.

The bug:

In general, execution should be able to evict storage memory when the total storage memory usage is greater than maxMemory * spark.memory.storageFraction. Due to a bug, however, Spark might wind up evicting no storage memory in certain cases where the storage memory usage was between maxMemory * spark.memory.storageFraction and maxMemory. For example, here is a regression test which illustrates the bug:

    val maxMemory = 1000L
    val taskAttemptId = 0L
    val (mm, ms) = makeThings(maxMemory)
    // Since we used the default storage fraction (0.5), we should be able to allocate 500 bytes
    // of storage memory which are immune to eviction by execution memory pressure.

    // Acquire enough storage memory to exceed the storage region size
    assert(mm.acquireStorageMemory(dummyBlock, 750L, evictedBlocks))
    assertEvictBlocksToFreeSpaceNotCalled(ms)
    assert(mm.executionMemoryUsed === 0L)
    assert(mm.storageMemoryUsed === 750L)

    // At this point, storage is using 250 more bytes of memory than it is guaranteed, so execution
    // should be able to reclaim up to 250 bytes of storage memory.
    // Therefore, execution should now be able to require up to 500 bytes of memory:
    assert(mm.acquireExecutionMemory(500L, taskAttemptId, MemoryMode.ON_HEAP) === 500L) // <--- fails by only returning 250L
    assert(mm.storageMemoryUsed === 500L)
    assert(mm.executionMemoryUsed === 500L)
    assertEvictBlocksToFreeSpaceCalled(ms, 250L)

The problem relates to the control flow / interaction between StorageMemoryPool.shrinkPoolToReclaimSpace() and MemoryStore.ensureFreeSpace(). While trying to allocate the 500 bytes of execution memory, the UnifiedMemoryManager discovers that it will need to reclaim 250 bytes of memory from storage, so it calls StorageMemoryPool.shrinkPoolToReclaimSpace(250L). This method, in turn, calls MemoryStore.ensureFreeSpace(250L). However, ensureFreeSpace() first checks whether the requested space is less than maxStorageMemory - storageMemoryUsed, which will be true if there is any free execution memory because it turns out that MemoryStore.maxStorageMemory = (maxMemory - onHeapExecutionMemoryPool.memoryUsed) when the UnifiedMemoryManager is used.

The control flow here is somewhat confusing (it grew to be messy / confusing over time / as a result of the merging / refactoring of several components). In the pre-Spark 1.6 code, ensureFreeSpace was called directly by the MemoryStore itself, whereas in 1.6 it's involved in a confusing control flow where MemoryStore calls MemoryManager.acquireStorageMemory, which then calls back into MemoryStore.ensureFreeSpace, which, in turn, calls MemoryManager.freeStorageMemory.

The solution:

The solution implemented in this patch is to remove the confusing circular control flow between MemoryManager and MemoryStore, making the storage memory acquisition process much more linear / straightforward. The key changes:

Remove a layer of inheritance which made the memory manager code harder to understand (5384117).
Move some bounds checks earlier in the call chain (13ba7ad).
Refactor ensureFreeSpace() so that the part which evicts blocks can be called independently from the part which checks whether there is enough free space to avoid eviction (7c68ca0).
Realize that this lets us remove a layer of overloads from ensureFreeSpace (eec4f6c).
Realize that ensureFreeSpace() can simply be replaced with an evictBlocksToFreeSpace() method which is called after we've already figured out how much memory needs to be reclaimed via eviction; (2dc842a).

Along the way, I fixed some problems with the mocks in MemoryManagerSuite: the old mocks would unconditionally report that a block had been evicted even if there was enough space in the storage pool such that eviction would be avoided.

I also fixed a problem where StorageMemoryPool._memoryUsed might become negative due to freed memory being double-counted when excution evicts storage. The problem was that StorageMemoryPoolshrinkPoolToFreeSpace would decrement _memoryUsed even though StorageMemoryPool.freeMemory had already decremented it as each evicted block was freed. See SPARK-12189 for details.

This reduces coupling between failed tests.

…o be dropped. Previously, ensureFreeSpace() might end up not dropping blocks if the total storage memory pool usage was less than the maximum possible storage pool usage.

SparkQA · 2015-12-07T05:54:46Z

Test build #47251 has finished for PR 10170 at commit 0eac7da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public abstract static class PrefixComputer\n

JoshRosen · 2015-12-07T07:58:09Z

I think that the test failures in BlockManagerSuite are related to the semantics of maxNumBytesToFree in StaticMemoryManager.acquireUnrollMemory.

Previously, it looks like maxNumBytesToFree was the amount of memory that we'd try to free in the memoryStore.ensureFreeSpace() call but due to the bug fixed in this patch I believe that we'd end up in the case where maxNumBytesToFree is less than the amount of free storage memory, so ensureFreeSpace() wouldn't evict anything even though we actually need numBytesToAcquire > maxNumBytesToFree bytes of memory.

After the fixes implemented here, we'll first claim as much free storage memory as possible, then subtract that from our memory goal and request the remaining memory via spilling. As a result, we are more prone to evict, which might be throwing off the original test case (it's a little tricky to say due to size estimation; I'll try to see if I can decouple that via mocking in order to make the test easier to reason about).

One minor question of semantics: up until now (and still) it looks like maxNumBytesToFree has been "the maximum amount of memory to attempt to free up via spilling" and not "an upper bound on the amount of memory that will be spilled in response to a request". I believe that this might be the right interpretation, since the latter interpretation would mean that large cached blocks could never be evicted by small unroll requests.

@andrewor14, is the original idea behind spark.storage.unrollFraction that it places an upper limit on the amount of total storage memory that can be used for unrolling? If that's the case, we can simplify StorageMemoryPool.acquireMemory to only accept a single numBytesToAcquire and cap the size of requests at min(spark.storage.unrollFraction * maxStorageMemory - currentUnrollMemory, requestedUnrollMemory).

andrewor14 · 2015-12-08T18:39:57Z

core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala

+    if (numBytesToAcquire > memoryFree && maxNumBytesToFree > 0) {
+      val additionalMemoryRequired = numBytesToAcquire - memoryFree
+      memoryStore.evictBlocksToFreeSpace(
+        Some(blockId), Math.min(maxNumBytesToFree, additionalMemoryRequired), evictedBlocks)


can we use math.min like we do in other places

actually, to improve readability a little:

val additionalMemoryRequired = ... val numBytesToFree = math.min(maxNumBytesToFree, additionalMemoryRequired) memoryStore.evictBlocksToFreeSpace(Some(blockId), numBytesToFree, evictedBlocks)

andrewor14 · 2015-12-08T23:17:43Z

@JoshRosen despite the number of comments I left I think this patch looks good. I did a close review to verify its correctness and that the two bugs were real issues. On the side I will investigate the test failures and hopefully get this merged soon.

This commit also adds a regression test for SPARK-12189 through the existing test "execution evicts storage" by adding an assert.

Rewrite some tests after changes in eviction

…est.

SparkQA · 2015-12-09T02:14:44Z

Test build #47378 has finished for PR 10170 at commit d182bdc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-09T02:25:56Z

Test build #47380 has finished for PR 10170 at commit f6fb406.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-09T02:28:30Z

Note to self RE: unroll fraction (just to be precise):

At a high-level, the StaticMemoryManager a request for unroll memory will be able to obtain more memory if:

there is free storage memory, or
currentUnrollMemory / maxStorageMemory < spark.memory.unrollFraction.

An unroll request can only evict blocks if finalUnrollMemory / maxStorageMemory < spark.memory.unrollFraction, where finalUnrollMemory is the amount of unroll memory that would be used after the eviction was performed.

SparkQA · 2015-12-09T03:17:19Z

Test build #47382 has finished for PR 10170 at commit e2090d1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-09T03:43:37Z

Test build #47383 has finished for PR 10170 at commit a43ed34.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Before this commit, unrolling would evict too many blocks, resulting in test failures in BlockManagerSuite. The root cause is that we used `maxUnrollMemory` as a cap for the extra amount of memory to evict for unrolling, which is incorrect. Instead, we should use it as a cap for the total amount of unroll memory and calculate the amount of memory to evict from there. The goal of this commit is to preserve the old behavior (in 1.5) as much as possible. This can be seen from the fact that BlockManagerSuite now passes without any modifications.

One way to fix the tests

andrewor14 · 2015-12-09T19:19:18Z

LGTM, I'll merge this once tests pass.

JoshRosen · 2015-12-09T19:22:26Z

Updated PR description to remove the following text from the end (which will be incorporated into separate PRs, most likely):

TODOs

Add stronger assertions or a dedicated regression test for the _memoryUsed < 0 bug, which was uncovered while testing this patch and a related fix for SPARK-12155.
See whether we can now remove the confusing MemoryStore.maxMemory method.

/cc @andrewor14 @zsxwing @yhuai @rxin

SparkQA · 2015-12-09T19:36:27Z

Test build #47433 has finished for PR 10170 at commit 7e9e191.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-12-09T19:39:26Z

Merging into master and 1.6!!

…execution This patch fixes a bug in the eviction of storage memory by execution. ## The bug: In general, execution should be able to evict storage memory when the total storage memory usage is greater than `maxMemory * spark.memory.storageFraction`. Due to a bug, however, Spark might wind up evicting no storage memory in certain cases where the storage memory usage was between `maxMemory * spark.memory.storageFraction` and `maxMemory`. For example, here is a regression test which illustrates the bug: ```scala val maxMemory = 1000L val taskAttemptId = 0L val (mm, ms) = makeThings(maxMemory) // Since we used the default storage fraction (0.5), we should be able to allocate 500 bytes // of storage memory which are immune to eviction by execution memory pressure. // Acquire enough storage memory to exceed the storage region size assert(mm.acquireStorageMemory(dummyBlock, 750L, evictedBlocks)) assertEvictBlocksToFreeSpaceNotCalled(ms) assert(mm.executionMemoryUsed === 0L) assert(mm.storageMemoryUsed === 750L) // At this point, storage is using 250 more bytes of memory than it is guaranteed, so execution // should be able to reclaim up to 250 bytes of storage memory. // Therefore, execution should now be able to require up to 500 bytes of memory: assert(mm.acquireExecutionMemory(500L, taskAttemptId, MemoryMode.ON_HEAP) === 500L) // <--- fails by only returning 250L assert(mm.storageMemoryUsed === 500L) assert(mm.executionMemoryUsed === 500L) assertEvictBlocksToFreeSpaceCalled(ms, 250L) ``` The problem relates to the control flow / interaction between `StorageMemoryPool.shrinkPoolToReclaimSpace()` and `MemoryStore.ensureFreeSpace()`. While trying to allocate the 500 bytes of execution memory, the `UnifiedMemoryManager` discovers that it will need to reclaim 250 bytes of memory from storage, so it calls `StorageMemoryPool.shrinkPoolToReclaimSpace(250L)`. This method, in turn, calls `MemoryStore.ensureFreeSpace(250L)`. However, `ensureFreeSpace()` first checks whether the requested space is less than `maxStorageMemory - storageMemoryUsed`, which will be true if there is any free execution memory because it turns out that `MemoryStore.maxStorageMemory = (maxMemory - onHeapExecutionMemoryPool.memoryUsed)` when the `UnifiedMemoryManager` is used. The control flow here is somewhat confusing (it grew to be messy / confusing over time / as a result of the merging / refactoring of several components). In the pre-Spark 1.6 code, `ensureFreeSpace` was called directly by the `MemoryStore` itself, whereas in 1.6 it's involved in a confusing control flow where `MemoryStore` calls `MemoryManager.acquireStorageMemory`, which then calls back into `MemoryStore.ensureFreeSpace`, which, in turn, calls `MemoryManager.freeStorageMemory`. ## The solution: The solution implemented in this patch is to remove the confusing circular control flow between `MemoryManager` and `MemoryStore`, making the storage memory acquisition process much more linear / straightforward. The key changes: - Remove a layer of inheritance which made the memory manager code harder to understand (5384117). - Move some bounds checks earlier in the call chain (13ba7ad). - Refactor `ensureFreeSpace()` so that the part which evicts blocks can be called independently from the part which checks whether there is enough free space to avoid eviction (7c68ca0). - Realize that this lets us remove a layer of overloads from `ensureFreeSpace` (eec4f6c). - Realize that `ensureFreeSpace()` can simply be replaced with an `evictBlocksToFreeSpace()` method which is called [after we've already figured out](https://github.com/apache/spark/blob/2dc842aea82c8895125d46a00aa43dfb0d121de9/core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala#L88) how much memory needs to be reclaimed via eviction; (2dc842a). Along the way, I fixed some problems with the mocks in `MemoryManagerSuite`: the old mocks would [unconditionally](https://github.com/apache/spark/blob/80a824d36eec9d9a9f092ee1741453851218ec73/core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala#L84) report that a block had been evicted even if there was enough space in the storage pool such that eviction would be avoided. I also fixed a problem where `StorageMemoryPool._memoryUsed` might become negative due to freed memory being double-counted when excution evicts storage. The problem was that `StorageMemoryPoolshrinkPoolToFreeSpace` would [decrement `_memoryUsed`](7c68ca0#diff-935c68a9803be144ed7bafdd2f756a0fL133) even though `StorageMemoryPool.freeMemory` had already decremented it as each evicted block was freed. See SPARK-12189 for details. Author: Josh Rosen <[email protected]> Author: Andrew Or <[email protected]> Closes #10170 from JoshRosen/SPARK-12165. (cherry picked from commit aec5ea0) Signed-off-by: Andrew Or <[email protected]>

JoshRosen added 8 commits December 6, 2015 12:57

Reset ensureFreeSpaceCalled before each test.

012cb4b

This reduces coupling between failed tests.

Add regression test for storage eviction bug.

b519fe6

Add MemoryStore.freeSpaceForExecution() method, which forces blocks t…

7c68ca0

…o be dropped. Previously, ensureFreeSpace() might end up not dropping blocks if the total storage memory pool usage was less than the maximum possible storage pool usage.

Remove a layer of confusing inheritance.

5384117

Put fail-fast for non-fitting blocks earlier in call chain.

13ba7ad

Collapse ensureFreeSpace overloads

eec4f6c

Replace ensureFreeSpace() with evictBlocksToFreeSpace().

2dc842a

Update JIRA in test case.

0eac7da

JoshRosen changed the title ~~[SPARK-12165] Fix bug in eviction of storage memory by execution~~ [SPARK-12165][SPARK-12189] Fix bug in eviction of storage memory by execution Dec 8, 2015

JoshRosen changed the title ~~[SPARK-12165][SPARK-12189] Fix bug in eviction of storage memory by execution~~ [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution Dec 8, 2015

andrewor14 reviewed Dec 8, 2015
View reviewed changes

Rewrite unroll test after changes in eviction

b82ee14

andrewor14 mentioned this pull request Dec 8, 2015

Rewrite some tests after changes in eviction JoshRosen/spark#8

Merged

Andrew Or and others added 8 commits December 8, 2015 15:52

Clarify regression test for SPARK-12165

528cd85

This commit also adds a regression test for SPARK-12189 through the existing test "execution evicts storage" by adding an assert.

Merge pull request #8 from andrewor14/josh-unroll-tests

d182bdc

Rewrite some tests after changes in eviction

Incorporate Andrew's redability suggestions.

e48ef21

maxMemory -> maxStorageMemory in check for blocks that can never fit

994585f

Avoid double-add to evictedBlocks.

f6fb406

was -> were

52b544c

Add comments to explain potentially-confusing 1-byte block eviction t…

e2090d1

…est.

Push evictedBlocks checks into assertEvictBlocksToFreeSpaceNotCalled.

a43ed34

Andrew Or and others added 2 commits December 9, 2015 00:47

Merge pull request #9 from andrewor14/josh-unroll-tests

7e9e191

One way to fix the tests

asfgit closed this in aec5ea0 Dec 9, 2015

JoshRosen deleted the SPARK-12165 branch December 9, 2015 19:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution #10170

[SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution #10170

Uh oh!

JoshRosen commented Dec 7, 2015

Uh oh!

SparkQA commented Dec 7, 2015

Uh oh!

JoshRosen commented Dec 7, 2015

Uh oh!

andrewor14 Dec 8, 2015

Uh oh!

andrewor14 Dec 8, 2015

Uh oh!

andrewor14 commented Dec 8, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

JoshRosen commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

andrewor14 commented Dec 9, 2015

Uh oh!

JoshRosen commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

andrewor14 commented Dec 9, 2015

Uh oh!

Uh oh!

[SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution #10170

[SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution #10170

Uh oh!

Conversation

JoshRosen commented Dec 7, 2015

The bug:

The solution:

Uh oh!

SparkQA commented Dec 7, 2015

Uh oh!

JoshRosen commented Dec 7, 2015

Uh oh!

andrewor14 Dec 8, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 Dec 8, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Dec 8, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

JoshRosen commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

andrewor14 commented Dec 9, 2015

Uh oh!

JoshRosen commented Dec 9, 2015

TODOs

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

andrewor14 commented Dec 9, 2015

Uh oh!

Uh oh!