Skip to content

Persist Claude and Codex freshness digests across engine restarts - #1393

Merged
wesm merged 14 commits into
mainfrom
perf/claude-codex-stat-digest-skip
Aug 14, 2026
Merged

Persist Claude and Codex freshness digests across engine restarts#1393
wesm merged 14 commits into
mainfrom
perf/claude-codex-stat-digest-skip

Conversation

@wesm

@wesm wesm commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Persist file-stat digests for Claude and Codex so syncs after an engine restart
can skip unchanged transcripts without reopening and hashing them.

  • Digests cover size, mtime, and ctime; Codex also includes
    session_index.jsonl, so title-only changes still resync.
  • Digests are stored only after a committed write or a content-verified skip,
    and existing rows are backfilled on their first verified pass.
  • Remote imports remain content-hash based because copied filesystem timestamps
    are not a reliable freshness signal.

On a 55k-session archive, a fresh-process sweep fell from more than 30 minutes
to about 39 seconds. Warm-process sweeps remain behaviorally unchanged.

The main implementation is in internal/parser/stat_digest.go and
internal/sync/engine.go, with restart and index-change coverage in the parser
and sync integration tests.

@wesm
perf(sync): persist Claude and Codex freshness digests across restarts
d202a32 
Claude and Codex Fingerprint content-hashes the whole transcript, and
the in-memory verified-source gate only spares that cost from a
process's second pass onward. A daemon restart or a one-shot CLI sync
therefore re-reads and hashes every unchanged transcript — about 28GB
across ~54k files on one real archive — before a single skip fires.

Implement MultiFileStatHasher for both providers so the engine's
provider_freshness stat-digest short-circuit applies: a digest over
(size, mtime, ctime) tuples is stamped after verified outcomes and
persists in SQLite, letting a fresh engine skip unchanged sources with
two stats and one small DB read. The ctime term preserves in-place
rewrite detection, matching the verified-source gate's signature. The
Codex digest folds session_index.jsonl exactly like the gate's sidecar
signature, so an index-only title rename always breaks the digest, and
providerStatFreshnessMtime returns the index-folded effective mtime so
cold and warm cache keys stay on the same basis.
@roborev-ci

roborev-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

roborev: Combined Review (d202a32)

Medium

  • internal/parser/claude_provider.go:679 — Claude’s persisted digest is checked only after providerSingleSessionFresh, which reads and hashes the entire transcript before normally returning. After restart, the engine still performs a full-content hash, and legacy rows never backfill a freshness digest. Consult the persisted digest first, stamp it after a content-verified unchanged skip when missing, and test that computeFileHashPrefix is not called after restart.

  • internal/parser/codex_provider.go:981 — Existing Codex rows without a freshness digest return through cache/DB skip paths before stampProviderStatHashForConfirmedSource, so the digest is never backfilled. Index changes also invalidate every rollout digest, while unchanged-title rollouts return without refreshing it, causing unaffected archives to be rehashed after each restart. Persist the captured digest after every DB-confirmed unchanged return, including validated cache skips, and test upgrade-without-digest and multi-rollout index-change cases.

Reviewers: 2 done | Synthesis: codex, 11s | Total: 4m21s

@wesm
fix(sync): repair stat-digest gate scoping and backfill digests on skips
5b4b267 
The digest-currency check used path-only lookups, so a TraeX skip could
borrow a newer Codex row on the same rollout path and hide a project
repair. Scope every lookup to the provider's own stored-agent labels
(codebuff+freebuff for the relabeling Codebuff provider, the discovery
label elsewhere) and update the hasher-registration test for the agents
that now declare the capability.

Stamp the provider_freshness digest on every skip that verifies current
content against a stored current row: the Claude content-verified
single-session skip, the hash-validated cache skip, and the Codex-family
DB-fingerprint skip. Rows that predate the side-table now backfill on
their first confirmed-unchanged pass instead of re-hashing on every
fresh process, and a shared session_index.jsonl touch no longer leaves
every unaffected rollout with a permanently stale digest. The digest
gate also runs before the Claude single-session content guard so a
digest match skips without reading the transcript.
@roborev-ci

roborev-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

roborev: Combined Review (5b4b267)

Code is clean: the only reported findings were low severity, and no medium, high, or critical issues were identified.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 5m20s

wesm added 2 commits August 13, 2026 16:35
@wesm
Merge remote-tracking branch 'origin/main' into perf/claude-codex-sta…
71811f7 
…t-digest-skip

* origin/main:
  ci: isolate slow Windows Go packages (#1395)
  fix(activity): avoid repeated snapshot peer scans (#1394)
  Keep in-chunk Codex tool outputs on the incremental parse path (#1391)
  Stop re-parsing titled Codex sessions when session_index.jsonl is absent (#1389)
  fix(parser): match Codex fork turns to parents (#1384)
@wesm
fix(sync): keep remote imports content-hash arbitrated, sync with main
8a82be0 
Merge main and repair the newCodexSessionBuilder test call the #1384 +
#1391 merge left non-compiling (also fixed on main by #1397).

Withhold stat-digest staging, stamping, and consultation for Claude and
Codex-family sources under a pathRewriter: a remote import materializes
a fresh physical file whose mtime is copied from the remote and whose
ctime is the import clock, so a same-stat different-content re-download
can collide with a stored digest inside one coarse filesystem timestamp
tick and skip a real rewrite (caught by
TestSyncAllCodexPathRewriterSameStatRewriteUsesContentHash on Linux
runners). Remote freshness for these providers stays content-hash
arbitrated; Codebuff keeps its deliberate remote digest under the
logical key.
@roborev-ci

roborev-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

roborev: Combined Review (8a82be0)

Code changes are clean: no Medium, High, or Critical findings were reported.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 4m58s

@wesm
perf(sync): restore warm no-op allocation baseline
a2a9aaf 
Persisted digest hits returned before they could establish in-memory source
trust. Each warm pass repeated digest stats and database repair checks, which
pushed allocations over the benchmark gate.

Promote verified local state after a trusted digest match and consult that
state before computing another digest. Fresh engines retain the persisted
shortcut, while later passes return to the established warm path.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (a2a9aaf)

Changes look sound overall, but one medium-severity freshness bug could leave Codex titles stale after a transient index read failure.

Medium

  • Codex freshness may be stamped without successfully checking the title index — internal/sync/engine.go:8879, internal/sync/engine.go:8944

    LookupCodexThreadNameEntry collapses read and scan failures into ok=false, which the title check interprets as unchanged. If session_index.jsonl temporarily cannot be read after a rename, the new stat digest can still be persisted. Later syncs may then skip the title lookup, leaving the stored title stale until the index stat changes again.

    Preserve an error or verification status from the index lookup, and only stamp the digest after the index was successfully checked or confirmed absent.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 12m46s

@wesm
fix(sync): retry Codex title lookup after index errors
84e21ed 
A transient session index read or scan failure looked like a confirmed
absence. The engine could persist a new stat digest while retaining a stale
title, so later restarts stopped retrying the title lookup.

Keep transcript skips independent, but promote or persist freshness trust only
after the title index is readable or confirmed absent. This preserves the warm
path while ensuring transient metadata failures get another chance.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (84e21ed)

Medium

  • internal/sync/engine.go:8145, internal/parser/claude_provider.go:679 — Claude may emit multiple sessions from one DAG transcript, but the source digest is attached only to the first result and persisted as soon as that row succeeds. If another fork fails to write, the valid digest can cause later syncs to skip the source permanently, leaving the missing fork unretried. Persist the digest only after all results from the source commit without vetoes or retries, or disable Claude digest freshness until source-level completion is tracked.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 8m16s

@wesm
fix(sync): commit Claude freshness per source
3384663 
Claude transcripts can emit a main session and fork sessions from one file.
Stamping freshness after the first successful row could hide a failed fork
forever after a restart.

Keep existing members stale before writes. Promote every member and persist the
digest only after the complete source succeeds, so partial writes remain
retryable.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (3384663)

Changes need revision: two medium-severity synchronization issues could cause repeated rewrites and negate incremental freshness optimization.

Medium

  • internal/sync/engine.go:7844, internal/sync/engine.go:16020 — Intentionally skipped Claude DAG members are treated as failed writes. When an excluded or trashed transcript changes, active branches remain below the current data version, the digest is cleared, and subsequent syncs repeatedly reparse and rewrite them.

    • Fix: Distinguish intentional skips from write failures, excluding skipped members from demotion, completion counting, and promotion requirements. Add coverage for a changed DAG containing a permanently excluded or trashed fork.

  • internal/sync/engine.go:9004 — Incremental Claude/Codex writes discard preParseStatHash. The freshness digest remains stale after an append, forcing the next no-op sync or restart to hash the entire transcript before backfilling the digest.

    • Fix: Carry the staged digest through successful incremental results and persist it when the parser consumed the complete fingerprinted source, while retaining retry behavior for partial EOF records.

Reviewers: 2 done | Synthesis: codex, 12s | Total: 8m5s

@wesm
fix(sync): preserve freshness across skipped writes
f3fa5eb 
Treat excluded and trashed Claude DAG members as resolved without promoting or counting them as writes, so active branches can complete without repeated rewrites.

Persist complete-source incremental digests after the database commit, withhold them for partial EOF records, and invalidate source freshness when a trashed member is restored.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (f3fa5eb)

Medium-severity issue found in Claude DAG restore freshness handling.

Medium

  • internal/sync/engine.go:8868 — Restoring a Claude DAG fork marks only that fork stale and clears the digest, but providerSingleSessionFresh validates only the transcript’s stem/main session. If the main row is current and the file is unchanged, parsing is skipped and the digest is re-stamped, leaving the restored fork stale and potentially omitting changes made while it was trashed.
    • Suggested fix: Before accepting Claude’s single-session freshness gate, require every active session sharing the agent/path to have a current data version, or mark all active source siblings stale when restoring a DAG member.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 10m5s

@wesm
fix(sync): reparse restored Claude forks
0daad36 
A restored fork could remain stale when the unchanged transcript's main row satisfied Claude's early freshness shortcut. That shortcut could then restore the digest without parsing the fork.

Require every active row for the source path to have the current data version before accepting stem-based freshness.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (0daad36)

Changes requested: one medium-severity sync retry bug can keep valid sibling sessions perpetually stale.

Medium

  • internal/sync/engine.go:8222, internal/sync/engine.go:16058 — sourceNeedsRetry is applied to every result. A single CWD-vetoed member or per-result retry writes otherwise valid siblings at CurrentDataVersion()-1, causing multi-session sources to be reparsed on every sync.
    • Fix: Reserve source-wide retry state for source completion, digest, and cache decisions. Derive each write’s needsRetry from provider-wide failures and that session’s own retry state, applying the Claude DAG override separately.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 11m57s

@wesm
fix(sync): keep valid source siblings current
2d327ac 
A per-result retry or CWD veto was treated as retry state for every member emitted by the source. Valid siblings stayed below the current data version and were rewritten on later syncs.\n\nKeep source-wide retry state for completion and freshness decisions, while applying provider-wide and per-session retry state to each write.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (2d327ac)

Verdict: One medium-severity backend parity issue should be fixed before merge.

Medium

  • internal/postgres/store.go:466 — PostgreSQL restoration only clears deletion fields, while SQLite restoration also demotes data_version to force reparsing. Sessions restored through pg serve may therefore remain incorrectly marked current despite changes made while trashed.

    Suggested fix: Mirror SQLite’s freshness invalidation in PostgreSQL, or ensure PostgreSQL restores invalidate the corresponding SQLite source. Add a backend-parity restoration test.

Reviewers: 2 done | Synthesis: codex, 18s | Total: 12m44s

@wesm
fix(postgres): stale restored sessions for reparse
290736f 
PostgreSQL kept restored sessions at the current data version. A session could
therefore appear fresh even when its source changed while it was in trash.

Match SQLite's restore contract by invalidating the restored row's data
version. The next source sync can then refresh its content.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (290736f)

Medium-severity issue found in the persisted digest freshness gate.

Medium

  • internal/sync/engine.go:10762 — The digest gate uses GetDataVersionByAgentPath, which includes user-trashed rows. After a parser-version bump, a stale trashed Claude fork or Codex session permanently invalidates the persisted digest, causing every full sync or restart to repeatedly hash and parse an unchanged source. Check data versions only for active rows, treating an all-trashed source as resolved because restoration already clears its digest and marks the row stale. Add coverage for an old-version trashed member across an engine restart.

Reviewers: 2 done | Synthesis: codex, 11s | Total: 12m10s

@wesm
fix(sync): ignore trashed rows in digest repair gate
ccd6d7c 
A stale data version on a user-trashed session defeated persisted source
digests after every restart. Unchanged transcripts were hashed again even
though trashed rows do not require parser repair.

Check repair state only across active rows. An all-trashed source remains
resolved until restore invalidates its digest and marks its row stale.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (ccd6d7c)

Review found one medium-severity issue affecting restored S3 sessions.

Medium

  • Restored S3 sessions may remain stale due to skip-cache behavior — internal/db/sessions.go:4734, internal/sync/engine.go:8599

    Restoring a session marks it stale and clears provider_freshness, but an existing S3 skip-cache entry may return before processS3Session checks data_version. As a result, a restored S3 session can retain content changed while it was trashed until the source fingerprint changes or sync is forced.

    Suggested fix: Validate the session’s current data_version before accepting the generic cached skip, or bypass that cache and use shouldSkipFileWithPrefix. Add a regression test covering restoration with an existing cached S3 entry.

Reviewers: 2 done | Synthesis: codex, 19s | Total: 12m14s

@wesm
test(sync): cover restored S3 cache bypass
caba985 
Restored S3 sessions rely on their stale data version to override an otherwise
matching in-memory skip entry. This contract lacked direct end-to-end coverage,
so cache ordering could appear to preserve content changed while in trash.

Exercise restoration with unchanged object metadata and fingerprint. Confirm
that sync fetches the source, stores its changed message, and promotes the row
to the current data version.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (caba985)

Medium-severity freshness issue: unavailable change-time can allow rewritten session files to bypass content fingerprinting.

Medium

  • internal/parser/stat_digest.go:28 — The change-time availability flag is discarded. When ctime/change-time is unavailable, Claude and Codex persist a nonzero digest based only on size and mtime, so a same-size, mtime-preserving rewrite can bypass content fingerprinting indefinitely after restart.
    • Fix: Treat unavailable change-time on required files as an unverified digest and fall back to content fingerprinting instead of persisting or consulting it.

Reviewers: 2 done | Synthesis: codex, 12s | Total: 10m2s

@wesm
fix(sync): reject unverified stat digests
cbd7bd0 
Claude and Codex reduced an unavailable change-time to zero inside an otherwise
valid size and mtime tuple. A same-stat rewrite could therefore match a
persisted digest and skip content hashing after restart.

Treat the whole digest as unverified and reject its zero sentinel before the
freshness lookup. The stat shortcut now applies only when every existing
component supplies a reliable change-time.
@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (cbd7bd0)

No Medium, High, or Critical issues found.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 12m58s

@wesm
wesm merged commit 1f03473 into main Aug 14, 2026
23 checks passed
@wesm
wesm deleted the perf/claude-codex-stat-digest-skip branch August 14, 2026 17:22
@wesm wesm mentioned this pull request Aug 14, 2026
Merged
wesm added a commit that referenced this pull request Aug 14, 2026
@wesm
fix(sync): bound startup reconciliation and expose daemon identity (#…
7469e87 
…1413)

Warm startup reconciliation could consume several CPU cores for tens of
minutes on large archives. The agent-scoped freshness query shape introduced
in #1349 allowed SQLite to choose the broad agent index, so each discovered
source could scan every stored session for that provider. These lookups now
require the existing source-path index, keeping the reconciliation path and
the digest-backed freshness path from #1393 bounded by the matching source
rows without adding an index migration.

Long startup is now identifiable and controllable before runtime publication.
Startup state includes the daemon build version and operating-system process
create time; status displays that version, while stop and restart can safely
target the exact starting process. Managed Caddy identity is published at
launch so forced startup shutdown also cleans up the proxy. Legacy startup
snapshots without process identity remain protected from signaling.

This is independent of #1374, which changes provider controls and watcher
scheduling but does not change these database lookups or startup-state
lifecycle behavior.


Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
@wesm wesm mentioned this pull request Aug 15, 2026
Open
wesm added a commit to salmonumbrella/agentsview that referenced this pull request Aug 16, 2026
@wesm
Merge remote-tracking branch 'origin/main' into feat/hosted-raw-custo…
4ea3f0b 
…dy-core

* origin/main: (25 commits)
  ci: retry failed Docker image builds (kenn-io#1437)
  Speed up HTTP sync by processing only changed sessions (kenn-io#1414)
  chore: remove dead code and unused frontend exports (kenn-io#1434)
  fix(parser): populate Kimi session cwd (kenn-io#1427)
  feat(frontend): add raw and formatted tool output display (kenn-io#1424)
  feat(insights): support OpenAI-compatible endpoints (kenn-io#1430)
  fix(parser): support legacy Zed thread schemas (kenn-io#1429)
  perf(signals): index duplicate prompt comparisons (kenn-io#1425)
  fix(config): honor explicit empty agent directory arrays (kenn-io#1423)
  feat(parser): add DeepSeek Harness session support (kenn-io#1402)
  test(sync): wait for archive audit stall state (kenn-io#1422)
  chore(deps): update github actions dependencies (kenn-io#1421)
  Tombstone stored Claude sessions a complete full parse no longer emits (kenn-io#1392)
  fix(sync): report stalled syncs as unhealthy (kenn-io#1419)
  Reduce idle CPU and let users turn off unused providers (kenn-io#1374)
  fix(serve): survive dual-stack port collisions at startup (kenn-io#1406)
  fix(sync): bound startup reconciliation and expose daemon identity (kenn-io#1413)
  fix(deps): update module golang.org/x/mod to v0.40.0 [security] (kenn-io#1400)
  Persist Claude and Codex freshness digests across engine restarts (kenn-io#1393)
  fix(sync): add lifecycle logging (kenn-io#1399)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wesm