Skip to content

Commit 1e42244

Browse files
macvincentfacebook-github-bot
authored andcommitted
Introduce JK to Disable Memory Reallocation Optimization
Summary: When implementing the stream chunker, we anticipated that stream buffers after chunking will end up growing to the size that previously triggered chunking. As a tradeoff between minimizing reallocations (for performance) and actually releasing memory (to relieve memory pressure), we heuristically determine the new buffer capacity for each stream. The issue with this optimization is that it conflicts with the rest of our memory tracking logic since we now have retained memory in the memory pool that is not accounted for. We now through local testing that disabling this optimization leads to better memory pressure relief. In this diff, we introduce a JK, [dwio/nimble_chunking:disable_memory_reallocation_optimization](https://www.internalfb.com/intern/justknobs/?name=dwio%2Fnimble_chunking#disable_memory_reallocation_optimization), that will be enabled just for DISCO experiments. This will help us understand the full impact of this optimization and whether it should be retained. Differential Revision: D87494427
1 parent 93a536d commit 1e42244

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

dwio/nimble/velox/StreamChunker.h

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#pragma once
1818

1919
#include "dwio/nimble/velox/StreamData.h"
20+
#include "justknobs/JustKnobProxy.h"
2021

2122
namespace facebook::nimble {
2223
namespace detail {
@@ -32,6 +33,12 @@ uint64_t getNewBufferCapacity(
3233
uint64_t maxChunkSize,
3334
uint64_t currentCapacityCount,
3435
uint64_t requiredCapacityCount) {
36+
static facebook::jk::BooleanKnob disableMemoryReallocationOptimization(
37+
"dwio/nimble_chunking:disable_memory_reallocation_optimization");
38+
if (disableMemoryReallocationOptimization()) {
39+
return requiredCapacityCount;
40+
}
41+
3542
const auto maxChunkElementCount = maxChunkSize / sizeof(T);
3643
if (currentCapacityCount < maxChunkElementCount) {
3744
return std::max<uint64_t>(

0 commit comments

Comments
 (0)