Skip to content

fix(sync): report stalled syncs as unhealthy - #1419

Merged
wesm merged 8 commits into
mainfrom
kenn-forge/issue-1403-sync-blocked-by-an-unresponsive-network-mount-shows-syncing-forever
Aug 15, 2026
Merged

fix(sync): report stalled syncs as unhealthy#1419
wesm merged 8 commits into
mainfrom
kenn-forge/issue-1403-sync-blocked-by-an-unresponsive-network-mount-shows-syncing-forever

Conversation

@wesm

@wesm wesm commented Aug 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Fixes #1403.

A failed network mount can leave the daemon alive while a sync pass is blocked
inside a filesystem call. Ping now preserves ok: true for daemon identity but
reports healthy: false with stalled sync details after five minutes without
progress. Sync status exposes the same timestamps and stalled flag.

In-process and worker-backed passes publish progress through the parent daemon.
Worker resyncs and daily archive audits are visible before their first worker
event and relay subsequent updates. Watcher batches acquire sync ownership
before rename planning, source preparation, discovery, or parsing. Coordinated
syncs clear progress after signal flushing and before PostgreSQL or DuckDB mirror
callbacks, so post-sync work is not mislabeled as stalled. New foreground sync
requests still fail promptly while another pass owns the engine; background work
remains serialized.

This detects the blocked state rather than attempting to cancel an uninterruptible
filesystem call. Operators can recognize and recover an unhealthy daemon without
clients treating the live process as missing and starting a duplicate.

wesm added 2 commits August 14, 2026 19:38
@wesm
fix(sync): report stalled syncs as unhealthy
eae7d14 
A failed network mount can block a filesystem call while the daemon remains
alive. Operators need to distinguish that state from a sync that is still
making progress without breaking the ping identity contract used to find and
control the existing daemon.

Mark active passes as stalled after five minutes without progress and expose
that state through ping and sync status. Keep ping identity valid so recovery
does not start a second daemon. Reject new foreground syncs promptly when an
existing pass owns the engine instead of waiting behind the blocked call.
@wesm
fix(sync): cover watcher batch reconciliation health
8562a2f 
Watcher push batches enter reconciliation through the shared locked helper.
They need to publish progress before discovery for the same reason as scheduled
passes: a blocked filesystem traversal must be visible in daemon health no
matter which reconciliation entry point reached it.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (8562a2f)

The changes are generally sound, but one medium-severity health-reporting gap remains.

Medium

  • internal/sync/engine.go:1350 — Changed-path syncs publish no progress before the first worker result. If the first parser in a path-only watcher batch hangs, CurrentProgress remains empty indefinitely and /api/ping incorrectly reports healthy: true. Publish initial syncing progress before startWorkers, and add a regression test with a blocked first changed-path parse.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 3m35s

@wesm
fix(sync): expose blocked changed-path parsing
684f395 
Watcher path batches can block before their first parser returns. Publish
initial progress before starting workers so daemon health can mark that work
stalled instead of remaining healthy.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (684f395)

Overall verdict: Two medium-severity sync health-reporting gaps remain; no security issues were found.

Medium

  • internal/sync/engine.go:1223 — Post-sync work can be falsely reported as a stalled sync.
    SyncThenRun, SyncThenRunWithRebuild, and SyncWatchBatchThenRun retain the final sync progress while executing potentially long PostgreSQL/DuckDB push callbacks. After five minutes, /api/ping may report an unhealthy stalled sync even though post-sync work is progressing, including phase: "done", stalled: true. Clear sync progress after synchronization and signal flushing but before invoking arbitrary post-sync work, or track that work with its own progress and heartbeat.

  • internal/sync/engine.go:1361 — Changed-path preparation can block without being reflected in health status.
    SyncPathsContext and SyncWatchBatchThenRun call prepareChangedPathSync before progress tracking begins. Its Lstat and provider classification operations may block on a failed network mount while /api/ping incorrectly reports the daemon as healthy. Establish ownership-safe progress before preparation, or move preparation into the serialized tracked pass, and add coverage for blocked stat/classification operations.

Reviewers: 2 done | Synthesis: codex, 13s | Total: 5m51s

@wesm
fix(sync): bound sync health to sync work
07dec13 
Changed-path preparation can block before the engine owns or reports the pass.
Post-sync mirror callbacks can outlive completed progress and make healthy work
look stalled.

Track preparation only after acquiring sync ownership, and end sync progress
before mirror callbacks. This keeps daemon health aligned with actual sync work
without weakening serialization.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (07dec13)

Medium findings remain: health reporting misses two sync phases, allowing stalled work to appear healthy.

Medium

  • internal/server/huma_routes_health.go:31 — Production full resyncs run in runWorkerResyncBuild, but its progress relay only forwards to the HTTP callback and never updates the parent engine. As a result, /api/ping can report healthy: true with no sync state during a stalled worker-backed resync. Mirror the sync-worker instrumentation: publish initial progress, relay worker progress through engine.UpdateProgress, and defer engine.FinishProgress.

  • internal/sync/watch_batch_sync.go:275 — Watch-batch planning occurs before the exclusive lock and before progress is published. Because planWatchBatch performs os.Stat and archive queries for unknown rename events, a block there remains invisible to health reporting. Move planning into the owned sync pass, publish discovering progress beforehand, and clear progress on planning errors.

Reviewers: 2 done | Synthesis: codex, 12s | Total: 6m19s

@wesm
fix(sync): track worker resyncs and watch planning
d311a34 
Worker resyncs and unknown-rename planning can block before daemon health
sees any active work. Operators need those owned phases to age into the same
stalled state as in-process syncs.

Relay resync worker progress through the parent engine and plan production
watcher batches only after acquiring sync ownership. Clear both records on
every exit so completed work does not remain unhealthy.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (d311a34)

Changes need one medium-severity fix before approval.

Medium

  • cmd/agentsview/worker_pass.go:184, cmd/agentsview/main.go:2715 — The daily archive audit calls runWorkerWritePass without a progress callback and emits no reconciliation progress. If the audit blocks while holding the exclusive sync lock, /api/ping can continue reporting healthy: true. Publish and defer-clear parent-side progress around the audit worker pass, optionally relay reconciliation progress, and test a blocked audit through CurrentProgress.

Reviewers: 2 done | Synthesis: codex, 11s | Total: 7m13s

@wesm
fix(sync): expose stalled archive audits
9c196ee 
Daily archive audits can block while holding exclusive sync ownership without
publishing progress. Daemon health then looks healthy even though all other
sync work is blocked.

Track the direct worker pass after it acquires ownership and relay child
updates. Clear the record on every exit so audit completion does not leave
stale health state.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (9c196ee)

Medium: The audit progress callback is not forwarded, so long-running active audits may be incorrectly reported as stalled and unhealthy.

  • cmd/agentsview/sync_worker.go:199 — ReconcileWatchRootsWithStats is called without onProgress. Plumb the callback through the reconciliation path and add a worker-level test verifying real audit progress is forwarded.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 6m0s

@wesm
fix(sync): forward archive audit progress
b0b299d 
Daily audit workers can continue reconciling sessions without emitting
heartbeats. The parent then ages the initial audit marker into a false stalled
state even while work advances.

Forward reconciliation discovery and page progress through the worker protocol.
This keeps daemon health current for active audits while preserving stalled
detection when work stops.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (b0b299d)

No issues found.

Reviewers: 2 done | Synthesis: codex | Total: 6m0s

@wesm
test(sync): wait for stalled progress transition
16466cb 
Windows can return the same clock tick immediately after a progress update.
Tests that use a one-nanosecond stall interval can therefore observe active
progress before it has aged into the stalled state.

Keep the simulated sync work blocked and wait for the real stalled transition.
This preserves the health contract assertions without depending on platform
clock granularity.
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (16466cb)

No issues found.

Reviewers: 2 done | Synthesis: codex | Total: 5m51s

@wesm
wesm merged commit a5cb40f into main Aug 15, 2026
26 checks passed
@wesm
wesm deleted the kenn-forge/issue-1403-sync-blocked-by-an-unresponsive-network-mount-shows-syncing-forever branch August 15, 2026 17:16
wesm added a commit to salmonumbrella/agentsview that referenced this pull request Aug 16, 2026
@wesm
Merge remote-tracking branch 'origin/main' into feat/hosted-raw-custo…
4ea3f0b 
…dy-core

* origin/main: (25 commits)
  ci: retry failed Docker image builds (kenn-io#1437)
  Speed up HTTP sync by processing only changed sessions (kenn-io#1414)
  chore: remove dead code and unused frontend exports (kenn-io#1434)
  fix(parser): populate Kimi session cwd (kenn-io#1427)
  feat(frontend): add raw and formatted tool output display (kenn-io#1424)
  feat(insights): support OpenAI-compatible endpoints (kenn-io#1430)
  fix(parser): support legacy Zed thread schemas (kenn-io#1429)
  perf(signals): index duplicate prompt comparisons (kenn-io#1425)
  fix(config): honor explicit empty agent directory arrays (kenn-io#1423)
  feat(parser): add DeepSeek Harness session support (kenn-io#1402)
  test(sync): wait for archive audit stall state (kenn-io#1422)
  chore(deps): update github actions dependencies (kenn-io#1421)
  Tombstone stored Claude sessions a complete full parse no longer emits (kenn-io#1392)
  fix(sync): report stalled syncs as unhealthy (kenn-io#1419)
  Reduce idle CPU and let users turn off unused providers (kenn-io#1374)
  fix(serve): survive dual-stack port collisions at startup (kenn-io#1406)
  fix(sync): bound startup reconciliation and expose daemon identity (kenn-io#1413)
  fix(deps): update module golang.org/x/mod to v0.40.0 [security] (kenn-io#1400)
  Persist Claude and Codex freshness digests across engine restarts (kenn-io#1393)
  fix(sync): add lifecycle logging (kenn-io#1399)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Sync blocked by an unresponsive network mount shows "syncing" forever

1 participant

@wesm