feat(cache): make FileCachePool thread-safe for multi-vCPU access - #1555
Merged
Conversation
xiaoyang-hhh
marked this pull request as ready for review
July 16, 2026 07:52
xiaoyang-hhh
marked this pull request as draft
July 16, 2026 09:25
xiaoyang-hhh
force-pushed
the
cache/multi-vcpu
branch
from
July 17, 2026 02:09
6c948f2 to
777ef6a
Compare
Guard all FileCachePool metadata (fileIndex_, lru_, cold tiers, totalUsed_, tuning state) with a coarse photon::mutex (m_lock_) so one pool can be shared across multiple photon vCPUs (OS threads). Invariants: - m_lock_ is held only across in-memory ops; never across open()/do_open() or forceRecycle()/eviction(), keeping the lock order rw_lock -> m_lock_ one-way (avoids ABBA with ObjectCache's per-item mutex and non-reentrant self-deadlock). - eviction/evict snapshot a victim under m_lock_, release it, do the I/O (open()+WLOCK truncate), then re-lock to finalize. - the write path accounts size under the store rw_lock (updateSpace fstats under m_lock_) so it can't drift against eviction's WLOCK+truncate; forceRecycle() is deferred to do_pwritev2 after rw_lock is released. - running_/exit_/isFull_ and LruEntry::truncate_done become std::atomic. Add a multi-vCPU concurrency stress test (CachePool.concurrent_stress). QuotaFilePool is left unchanged and documented as not-yet-thread-safe (it is currently unwired: the factory always builds a plain FileCachePool). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xiaoyang-hhh
force-pushed
the
cache/multi-vcpu
branch
from
July 17, 2026 03:00
777ef6a to
890e4bc
Compare
xiaoyang-hhh
marked this pull request as ready for review
July 20, 2026 06:24
EricHuangqx
approved these changes
Jul 22, 2026
xiaoyang-hhh
added a commit
that referenced
this pull request
Jul 27, 2026
) * feat(cache): make FileCachePool thread-safe for multi-vCPU access Guard all FileCachePool metadata (fileIndex_, lru_, cold tiers, totalUsed_, tuning state) with a coarse photon::mutex (m_lock_) so one pool can be shared across multiple photon vCPUs (OS threads). Invariants: - m_lock_ is held only across in-memory ops; never across open()/do_open() or forceRecycle()/eviction(), keeping the lock order rw_lock -> m_lock_ one-way (avoids ABBA with ObjectCache's per-item mutex and non-reentrant self-deadlock). - eviction/evict snapshot a victim under m_lock_, release it, do the I/O (open()+WLOCK truncate), then re-lock to finalize. - the write path accounts size under the store rw_lock (updateSpace fstats under m_lock_) so it can't drift against eviction's WLOCK+truncate; forceRecycle() is deferred to do_pwritev2 after rw_lock is released. - running_/exit_/isFull_ and LruEntry::truncate_done become std::atomic. Add a multi-vCPU concurrency stress test (CachePool.concurrent_stress). QuotaFilePool is left unchanged and documented as not-yet-thread-safe (it is currently unwired: the factory always builds a plain FileCachePool). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * check total used in eviction loop --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
lihuiba
pushed a commit
that referenced
this pull request
Jul 27, 2026
) (#1571) * feat(cache): make FileCachePool thread-safe for multi-vCPU access Guard all FileCachePool metadata (fileIndex_, lru_, cold tiers, totalUsed_, tuning state) with a coarse photon::mutex (m_lock_) so one pool can be shared across multiple photon vCPUs (OS threads). Invariants: - m_lock_ is held only across in-memory ops; never across open()/do_open() or forceRecycle()/eviction(), keeping the lock order rw_lock -> m_lock_ one-way (avoids ABBA with ObjectCache's per-item mutex and non-reentrant self-deadlock). - eviction/evict snapshot a victim under m_lock_, release it, do the I/O (open()+WLOCK truncate), then re-lock to finalize. - the write path accounts size under the store rw_lock (updateSpace fstats under m_lock_) so it can't drift against eviction's WLOCK+truncate; forceRecycle() is deferred to do_pwritev2 after rw_lock is released. - running_/exit_/isFull_ and LruEntry::truncate_done become std::atomic. Add a multi-vCPU concurrency stress test (CachePool.concurrent_stress). QuotaFilePool is left unchanged and documented as not-yet-thread-safe (it is currently unwired: the factory always builds a plain FileCachePool). * check total used in eviction loop --------- Co-authored-by: Xiaoyang Lu <luxiaoyang.lxy@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make a single
FileCachePoolsafe to use concurrently from multiple photon vCPUs (OS threads).How
photon::mutex m_lock_guarding all pool metadata (fileIndex_,lru_, cold tiers,totalUsed_).rw_lock → m_lock_:m_lock_is held only across in-memory ops.eviction/evict: snapshot victim underm_lock_→ release → do I/O (open()+WLOCK truncate) → re-lock to finalize.rw_lock(updateSpacefstats underm_lock_), so accounting can't drift against eviction's WLOCK+truncate;forceRecycle()deferred todo_pwritev2after therw_lockis released.running_/exit_/isFull_andLruEntry::truncate_done→std::atomic.Tests
CachePool.concurrent_stress: writers vs evictors on a shared small file set across N vCPUs (max per-file rw_lock contention). Validates no crash/hang/metadata-corruption/totalUsed_drift.ossfs2: all related tests pass, no deadlock/crash.Not in scope
QuotaFilePoolis not made thread-safe; documented with a TODO.