fix(manager): quarantine damaged daemon records instead of aborting recovery - #780
Open
iaroslav-reflection wants to merge 2 commits into
Open
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #780 +/- ##
==========================================
+ Coverage 26.01% 26.33% +0.32%
==========================================
Files 133 133
Lines 12711 12712 +1
==========================================
+ Hits 3307 3348 +41
+ Misses 8998 8951 -47
- Partials 406 413 +7
🚀 New features to boost your workflow:
|
Contributor
Author
Contributor
Author
|
The failing |
Zephyrcf
reviewed
Jul 21, 2026
Comment on lines
+343
to
+346
| log.L.Errorf("Quarantining daemon %s: failed to reload configuration %s, %s", d.ID(), d.ConfigFile(""), err) | ||
| quarantined = append(quarantined, d) | ||
| //nolint:nilerr | ||
| return nil |
Contributor
There was a problem hiding this comment.
Suggested change
| log.L.Errorf("Quarantining daemon %s: failed to reload configuration %s, %s", d.ID(), d.ConfigFile(""), err) | |
| quarantined = append(quarantined, d) | |
| //nolint:nilerr | |
| return nil | |
| //nolint:nilerr | |
| return nil |
I think it's best to skip this part; the existence of these bad records won't actually cause any problems. Another scenario is accidentally deleting these configuration file directories, which means that the existing daemon will be deleted during the subsequent recovery operation. In reality, I only need to restore the corresponding configuration files. Therefore, skipping this part instead of deleting is a conservative and effective approach.
Contributor
Author
There was a problem hiding this comment.
Agreed, changed to skip without deleting. The daemon is still dropped from the in-memory cache since it's registered before the config load.
…ecovery recoverDaemons returned an error when a persisted fusedev daemon record's configuration file could not be reloaded, which aborts the whole WalkDaemons pass, fails Recover and therefore NewSnapshotter: the snapshotter exits on startup. Since nothing ever prunes daemon records, a single damaged record crash-loops the snapshotter on every subsequent restart until the database is repaired by hand. Such records are easy to acquire: the daemon record is persisted by createDaemon before the configuration is dumped to disk, so a crash or a full disk (the likeliest cause, since the blob cache shares the volume) in that window leaves a record without a config.json. A power loss can truncate an existing config the same way. Recovery must be fault-isolated per record: quarantine records whose configuration cannot be reloaded — log, skip, and prune them after the walk — and continue recovering everything else. A daemon without its configuration cannot be recovered or managed anyway; dropping the record keeps the next restart from tripping over it, and orphan process cleanup is already handled by the regular vestige clearing. Persisting the record only after its configuration is on disk is a worthwhile follow-up hardening; this change makes recovery robust to any such damage regardless of how it was produced. Signed-off-by: Iaroslav Geraskin <iaroslav@reflection.ai>
A missing config file does not prove the daemon behind the record is dead. Keep the record so a later restart can re-adopt the daemon once the config is restored; only drop the in-memory registration. Signed-off-by: Iaroslav Geraskin <iaroslav@reflection.ai>
iaroslav-reflection
force-pushed
the
fix/resilient-daemon-recovery
branch
from
July 21, 2026 18:41
c5f5f7b to
1252e2a
Compare
Contributor
Author
Contributor
Author
|
@imeoer ping — approved since July, good to merge? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A persisted fusedev daemon record whose
config.jsoncannot be reloaded maderecoverDaemonsreturn an error, aborting the wholeWalkDaemonspass and failingNewSnapshotter. Since daemon records are never pruned, one damaged record crash-loops the snapshotter on every restart until the boltdb is repaired by hand.Such records are easy to acquire:
createDaemonpersists the record before the configuration is dumped to disk, so a crash or a full disk (the likeliest cause, since the blob cache shares the volume) in that window leaves a record without a config file; a power loss can truncate an existing config the same way.flowchart LR A["record persisted,<br/>config.json missing"] --> B["restart → recovery walk"] B --> C["config reload fails →<br/>whole walk aborts"] C --> D["NewSnapshotter fails →<br/>snapshotter exits"] D --> B C --> E["💥 crash loop until manual<br/>boltdb surgery"]Make recovery fault-isolated per record: skip records whose configuration cannot be reloaded — log and continue recovering everything else. The record stays in the store, so restoring the config file lets a later restart re-adopt the daemon.
Persisting the record only after its configuration is on disk is a worthwhile follow-up hardening; this change makes recovery robust to such damage regardless of how it was produced.