fix(manager): surface liveness subscribe failures instead of reporting success - #778
Open
iaroslav-reflection wants to merge 1 commit into
Open
Conversation
Contributor
Author
iaroslav-reflection
force-pushed
the
fix/liveness-subscribe-errors
branch
from
July 20, 2026 16:49
2e42d0e to
a2664b9
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #778 +/- ##
==========================================
+ Coverage 25.42% 26.05% +0.63%
==========================================
Files 133 133
Lines 12639 12719 +80
==========================================
+ Hits 3213 3314 +101
+ Misses 9036 8998 -38
- Partials 390 407 +17
🚀 New features to boost your workflow:
|
Zephyrcf
approved these changes
Jul 21, 2026
| if closeErr := uc.Close(); closeErr != nil { | ||
| log.L.WithError(closeErr).Warnf("close liveness connection for daemon %s", id) | ||
| } | ||
| return |
Contributor
There was a problem hiding this comment.
Suggested change
| return | |
| return err |
Contributor
Author
There was a problem hiding this comment.
Thanks, applied!
…g success livenessMonitor.Subscribe armed the epoll subscription inside a syscall.RawConn.Control callback that assigned failures to the function's named error return -- which the enclosing `err = rawConn.Control(...)` statement then overwrote with Control's own nil result. A failed unix.SetNonblock or unix.EpollCtl was therefore logged, "Subscribe daemon ... liveness event" was still printed, and Subscribe returned nil. The caller then believes the daemon is monitored when it was never added to the epoll interest list: when that nydusd later dies no death event fires, so restart/failover recovery never starts and the daemon's containers are left with dead FUSE mounts -- with nothing in the logs explaining why. The dialed unix connection also leaked on this path. Introduce a controlFD helper that returns the first error from either Control itself or the callback, make Subscribe all-or-nothing (close the connection and register nothing on failure), and use the same helper in unsubscribe, where an EPOLL_CTL_DEL failure previously left a stale fd entry in the interest set that could collide with a future subscription reusing the same fd number. Signed-off-by: Iaroslav Geraskin <iaroslav@reflection.ai>
iaroslav-reflection
force-pushed
the
fix/liveness-subscribe-errors
branch
from
July 21, 2026 16:50
a2664b9 to
3479119
Compare
Contributor
Author
Contributor
Author
|
@imeoer ping — approved since July, good to merge? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Subscribearmed the epoll watch inside arawConn.Controlcallback that assigned failures to the named error return — which the enclosingerr = rawConn.Control(...)then overwrote with Control's own nil result. A failedSetNonblock/EpollCtlstill logged "Subscribe daemon ... liveness event" and returned nil.flowchart LR A["err = rawConn.Control(closure)"] --> B["closure: EpollCtl fails → sets err"] B --> C["Control returns nil →<br/>err overwritten"] C --> D["caller believes daemon is monitored"] D --> E["💥 nydusd dies → no death event →<br/>no restart/failover → ENOTCONN,<br/>logs claim monitoring was active"]The daemon was never added to the epoll interest list, so its death produces no event and restart/failover recovery never starts — with nothing in the logs explaining why. The dialed unix connection also leaked on this path.
Add a
controlFDhelper returning the first error from either Control or the callback, makeSubscribeall-or-nothing (close the connection and register nothing on failure), and use the helper inunsubscribe, where anEPOLL_CTL_DELfailure previously left a stale fd entry in the interest set that could collide with a future subscription reusing the same fd number.