Slab release duration

[vaavaav]

In the documentation, it says that:

Depending on the PoolResizer configuration and the amount of memory to be resized, cachelib might take a few minutes to hours to finish resizing the pool.

So, I have some questions:

1. I'm not seeing any way to set up the poolresizer from cachebench (i.e., there is no config field for that in any json file, nor on the respective code). I saw that poolRebalancer and poolResize share a lot of similarities. Can poolRebalancer replace for poolResizer? Or should I stick only to poolResizer for resizing slabs?

2. I've been plotting some slab release durations, with an average of 10 ms and std dev of 25+s ms, not near the expected values. This is likely because my configuration is too unrealistic and has a light workload. Is there any way I can set up cachebench to have slab releases take that much time?

3. Also, I'm not understanding very well how to get the current real size (i.e, the actual allocated and dedicated space for the respective pool) of each pool... Could you please point out what memory allocator method to use? For both current real size and current free space? This is important to me because I'm dynamically resizing the pools.

[jaesoo-fb]

Hi @vaavaav

1. As you know, the pool resizer is to resize/adjust the size of pools if those are set differently (at runtime) from what is initially configured. So, this is needed only if the application wants to resize the pool at runtime, while cachebench has no need to do so.

On the other hand, the pool rebalancer is to rebalance sizes between the allocation classes (as opposed to between pools for the pool resizer). The pool rebalancer is enabled by default with the default strategy. https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocatorConfig.h#L450

2. We don't control the slab release duration. For slab rebalancer and resizer, we configure/control the intervals between the rebalance/resize attempts and, for slab resizer, the number of slabs to be moved at each iteration.

3. Please refer to overLimit code which compares the configured size with actually used size.

[vaavaav]

1. Yeah, I saw that. I made a little change to cachebench to use and configure the pool resizer. Thank you for the insight tho

2. I know you can't force duration of slab releases. What I meant was if there was some cachebench setup that would be close to the ones where the "minutes to hours" slab release durations were seen.

3. I saw that aswell, I just find kind of confusing to see sizes of some KB but free sizes of MB. Why is there free space? Is this because the slab class is allocated and there is free space in it?

[jaesoo-fb]

2. Actually, we don't expect the slab release (for slab rebalancing in cachebench) took more than a minutes; we don't hold any handle reference which can block the release for that long with cachebench.

3. Could you provide more concrete numbers? What variables/metrics are you seeing some KB and free sizes of MB for? Also, please be advised that the slab size is 4MB.

[vaavaav]

2. You say that about resizing. If it's not the slab release that take that much time, then what else? https://cachelib.org/docs/Cache_Library_User_Guides/Partition_cache_into_pools/#1-manual-pool-resizing

3. I'll try to reply later or tomorrow with the config I'm using and the values I'm getting.

Thank you very much for your input!

[vaavaav]

Config I'm using (for now):

{
  "cache_config" : {
    "cacheSizeMB" : 512,
    "poolRebalanceIntervalSec" : 1,
    "allocFactor" : 2.5,
    "rebalanceMinSlabs": 5,

    "numPools" : 10,
    "poolSizes" : [0.1, 0.1, 0.1, 0.1, 0.1,0.1, 0.1, 0.1, 0.1, 0.1]
  },
  "test_config" : {
      "numOps" : 1000000,
      "numThreads" : 32,
      "numKeys" : 1000000,

      "keySizeRange" : [8, 8],
      "keySizeRangeProbability" : [1.0],

      "valSizeRange" : [128000, 256000, 512000, 1000000],
      "valSizeRangeProbability" : [0.25,0.25,0.25,0.25],

      "getRatio" : 0.15,
      "setRatio" : 0.8,
      "delRatio" : 0.05,
      "keyPoolDistribution": [0.1, 0.1, 0.1, 0.1, 0.1,0.1, 0.1, 0.1, 0.1, 0.1],
      "opPoolDistribution" : [0.1, 0.1, 0.1, 0.1, 0.1,0.1, 0.1, 0.1, 0.1, 0.1]
    }
}

Two notes:

• I changed cachebench to also enable poolResizer (using poolRebalanceIntervalSec for its interval and rebalanceMinSlabs as slabs per iteration)
• I'm using [8, 8] for keySizeRange because, for now, I only want keys with exactly 8 bytes of size. I was getting an error if I only used 8, because it needed to be an array.

(Excerpt from the) Results I'm getting with a custom logger:

[2023-03-28 14:17:09.404] [ca] [info] Collecting cache status
[2023-03-28 14:17:09.499] [ca] [debug] id=0 hits=38 size=2588556 free=50331648
[2023-03-28 14:17:09.499] [ca] [debug] id=1 hits=38 size=72190261 free=37748736
[2023-03-28 14:17:09.499] [ca] [debug] id=2 hits=36 size=362662 free=0
[2023-03-28 14:17:09.499] [ca] [debug] id=3 hits=32 size=148733 free=0
[2023-03-28 14:17:09.499] [ca] [debug] id=4 hits=33 size=144669355 free=62914560
[2023-03-28 14:17:09.499] [ca] [debug] id=5 hits=37 size=127591205 free=12582912
[2023-03-28 14:17:09.499] [ca] [debug] id=6 hits=31 size=24846198 free=8388608
[2023-03-28 14:17:09.499] [ca] [debug] id=7 hits=31 size=86484713 free=29360128
[2023-03-28 14:17:09.499] [ca] [debug] id=8 hits=34 size=13698200 free=0
[2023-03-28 14:17:09.499] [ca] [debug] id=9 hits=37 size=60096717 free=29360128
[2023-03-28 14:17:09.499] [ca] [info] Computing best configuration
[2023-03-28 14:17:09.499] [ca] [debug] Solution: 1->7 (delta:5889205)
[2023-03-28 14:17:09.499] [ca] [info] Enforcing solution
[2023-03-28 14:17:09.499] [ca] [debug] Giver new size = 66301056
[2023-03-28 14:17:09.499] [ca] [debug] Taker new size = 92373918
[2023-03-28 14:17:09.532] [ca] [info] Collecting cache status
[2023-03-28 14:17:09.710] [ca] [debug] id=0 hits=38 size=2588556 free=50331648
[2023-03-28 14:17:09.710] [ca] [debug] id=1 hits=38 size=66301056 free=37748736
[2023-03-28 14:17:09.710] [ca] [debug] id=2 hits=36 size=362662 free=0
[2023-03-28 14:17:09.710] [ca] [debug] id=3 hits=32 size=148733 free=0
[2023-03-28 14:17:09.710] [ca] [debug] id=4 hits=33 size=144669355 free=62914560
[2023-03-28 14:17:09.710] [ca] [debug] id=5 hits=37 size=127591205 free=16777216
[2023-03-28 14:17:09.710] [ca] [debug] id=6 hits=31 size=24846198 free=8388608
[2023-03-28 14:17:09.710] [ca] [debug] id=7 hits=31 size=92373918 free=37748736
[2023-03-28 14:17:09.710] [ca] [debug] id=8 hits=34 size=13698200 free=0
[2023-03-28 14:17:09.710] [ca] [debug] id=9 hits=37 size=60096717 free=29360128

Considering stats as getPoolStats(id), for some poolId:

• size = stats.pooUsableSize + stats.poolAdvisedSize
• free = stats.freeMemoryBytes()

[jaesoo-fb]

size = stats.pooUsableSize + stats.poolAdvisedSize

I think what you want here is just stats.poolSize

I saw that aswell, I just find kind of confusing to see sizes of some KB but free sizes of MB. Why is there free space? Is this because the slab class is allocated and there is free space in it?

The poolSize is configured size; e.g., (512MB - <slab header overhead>) * 0.1 = ~5MB. Note that this is in bytes and not aligned on the slab size. The allocated unit for each pool and allocation class is in slabs (i.e., multiples of 4MB). There could be free space if some allocations are not used.