Codex stale-thinking spinner persists after stream state desync (thread stale detection + targeted recovery results)

[g0udurix]

Reproducibility summary

• Date: 2026-02-26 (UTC)
• Scope: local sessions under ~/.codex/sessions, ~/.codex/archived_sessions, app logs under ~/Library/Logs/com.openai.codex.
• Detector is read-only for discovery; recovery attempts are targeted per-thread only.
• This issue is tracked with generated artifacts under:
  • /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221328Z/
• No raw conversation text or secrets are included in the artifacts.

Key stale indicators

• 019c9a7f-46f2-7e03-aceb-89afe89a581b => stale score=12.6, unmatched_starts=2, heartbeat_ms=4650543, in_progress_stream=true, marked_streaming=true, turn_not_found_count=1272.
• 019c489a-0eb0-7142-8c30-e92ced2cf74a => at-risk score=5.8, unmatched_starts=0, heartbeat_ms=8332044, in_progress_stream=true, marked_streaming=true, turn_not_found_count=0.

Recovery results from tooling

• Targeted conservative recovery (--recover) was executed per-thread after detection on the two above threads.
• For both threads, the app-server returned RPC error: thread not found during turn/interrupt.
• Recovery status was tracked as thread_not_found; operator guidance is to continue in a fresh thread using a compact handoff of last confirmed artifacts.

Observable loop behavior

• Thread UI remains in thinking/streaming indications after maybe_resume_success appears with latestTurnStatus=inProgress and markedStreaming=true.
• Turn not found for event and thread-stream-state-changed / handler warning patterns continue in logs around those thread IDs.
• This suggests a stream-state reconciliation or state-database migration/cleanup race rather than pure compute inactivity.

Ask

• Please investigate stream-state lifecycle for thread-stream-state-changed, maybe_resume_success, and Turn not found for event reconciliation.
• Confirm whether stale stream markers should be auto-cleared when referenced turns are no longer resolvable.

[g0udurix]

Follow-up from production-hardening run:

• Run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221428Z/
• Target threads:
  • 019c9a7f-46f2-7e03-aceb-89afe89a581b
  • 019c489a-0eb0-7142-8c30-e92ced2cf74a
• Tool classification:
  • stale and at-risk respectively (heartbeat > 1h, stream markers present)
• Recovery attempt (automated, thread-safe): thread_not_found for both IDs on turn/interrupt.
• Script now treats this as terminal thread_not_found state and emits operator fallback guidance rather than generic failure.
• Recommendation remains: continue in a fresh thread using compact handoff of last confirmed completed steps.

Artifacts in this run include:

• stale_threads.jsonl
• actions.json
• incident_summary.md
• redacted_session_map.csv
• openai_feedback_draft.md
• recovery_results.json

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• Codex stale-thinking thread spinner may persist after stream-state mismatch #12954

Powered by Codex Action

[g0udurix]

• Follow-up update from latest run:
  • Added phrase-signal escalation path in detector (you are stuck somewhere. stop and replan / Reprise: j’ai relancé un test MLX), and explicit per-thread recovery forcing for --recover-thread targets.
  • Latest run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221609Z/
  • 019c9ba7-e168-78a1-97f2-a188bc74d44a also now surfaces as targeted recovery candidate (with phrase-signal evidence), but app-server returns thread_not_found when interrupted.
  • Recovery outputs remain per-thread only, non-global; no unrelated threads are stopped.

[g0udurix]

Follow-up production-hardening result (run ):

• Script: [stale-think] stale_threads=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/stale_threads.jsonl
  [stale-think] actions=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/actions.json
  [stale-think] summary=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/incident_summary.md
  [stale-think] map=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/redacted_session_map.csv
  [stale-think] openai_feedback=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/openai_feedback_draft.md
  [stale-think] github_payload=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/github_issue_payload.json
  [stale-think] stale_count=2 total_threads=57
• Targets: , ,
• Detection outcome: / / phrase-linked candidate.
• Recovery outcome: all three return on with required operator path: or (fresh thread + compact handoff).
• Version matrix is now explicitly tracked and this run shows mismatch: vs ().
• Evidence artifacts in including recovery results, stale rows, and final report.
• Automation remains active: (, no global stop actions).

[g0udurix]

Follow-up production-hardening result (run 20260226T222100Z):

• Script: /Users/jeremie/.codex/scripts/codex_stale_thinking_monitor.py
• Targets: 019c9a7f-46f2-7e03-aceb-89afe89a581b, 019c489a-0eb0-7142-8c30-e92ced2cf74a, 019c9ba7-e168-78a1-97f2-a188bc74d44a
• Detection outcome: stale, at-risk, phrase-linked candidate.
• Recovery outcome: all three return thread_not_found on turn/interrupt with required operator path: resume? if still visible, otherwise start fresh thread with compact handoff.
• Version alignment: codex_cli 0.101.0 vs codex_app 0.105.0 (mismatch).
• Evidence artifacts: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222100Z/ (recovery_results, stale_threads, openai_feedback_draft, incident_summary).
• Automation remains active: /Users/jeremie/.codex/automations/codex-stale-think-watch/automation.toml (hourly, no global stop actions).

[g0udurix]

Updated follow-up packet (run 20260226T222239Z):

• Recovery remains blocked for primary stale candidate 019c9a7f-46f2-7e03-aceb-89afe89a581b and at-risk 019c489a-0eb0-7142-8c30-e92ced2cf74a in this execution window.
• Final RC package now in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222239Z/final_rc_report.md.
• Version mismatch continues: codex_cli 0.101.0 vs codex_app 0.105.0 (version_alignment: mismatch).
• This run adds explicit operator handoff language in tooling: Recovered thread: resume? or new (fresh thread handoff). Automation prompt updated accordingly in automations/codex-stale-think-watch/automation.toml.

[g0udurix]

Follow-up production-hardening run after CLI/App alignment:\n\n- Aligned CLI to app protocol version via
changed 2 packages in 435ms (now both report ).\n- Re-ran targeted recovery on: and .\n- Result remains terminal for both: app-server returns on ; status in .\n- Operator handoff remains explicit: if thread still visible in UI, otherwise to continue from compact handoff.\n- Artifacts for this run:\n - stale_threads.jsonl\n - actions.json\n - openai_feedback_draft.md\n - recovery_results.json\n - incident_summary.md\n - github_issue_payload.json\n\nCurrent blocker for closure is persisted stream-state artifact for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

[g0udurix]

Follow-up production-hardening run after CLI/App alignment:

• Aligned CLI to app protocol version via npm install -g @openai/codex@0.105.0 (now both report codex-cli 0.105.0).
• Re-ran targeted recovery on thread IDs:
  • 019c9a7f-46f2-7e03-aceb-89afe89a581b
  • 019c489a-0eb0-7142-8c30-e92ced2cf74a
• Result remains terminal for both: app-server returns thread not found on turn/interrupt; status thread_not_found in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/recovery_results.json.
• Operator handoff remains explicit: Recovered thread: resume? if thread still visible in UI, otherwise new for compact handoff in a fresh thread.
• Artifacts for this run:
  • stale_threads.jsonl
  • actions.json
  • openai_feedback_draft.md
  • recovery_results.json
  • incident_summary.md
  • github_issue_payload.json

Current blocker for closure is persisted stream-state artifacts for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

[g0udurix]

RC-ready summary posted in local packet:
/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/final_rc_report.md

This confirms both flagged threads are in final state:

• thread_not_found on targeted recovery attempts
• safe recovery mode required operator handoff only
• explicit prompt available: Recovered thread: resume? or new

CLI/App are now aligned at 0.105.0 (npm list -g @openai/codex --depth=0), so remaining blocker is stale/orphaned thread state in old session artifacts.

[g0udurix]

Follow-up (2026-02-27, production rollout + evidence):

We implemented and validated an automated stale-think containment path in local production.

What is now automated:

1. Detect stale threads from logs/transcripts using:

• unmatched task_started/task_complete
• inProgress+markedStreaming markers
• turn reconciliation errors ("thread not found", "Turn not found for event")
• heartbeat gap

2. Conservative per-thread recovery first (no global restart).

3. If recovery fails (thread_not_found / still stale), auto-relaunch to a fresh thread in Plan mode with continuity context and explicit ops guidance bootstrap.

4. Crash protection in the same loop:

• detects Chrome runaway memory by total RSS and peak process RSS
• optional peak-process termination to prevent system hangs

Operational evidence (sanitized):

• Source stale threads:
  • 019c489a-0eb0-7142-8c30-e92ced2cf74a
  • 019c9a7f-46f2-7e03-aceb-89afe89a581b
• Recovery attempts on stale source threads return thread_not_found even while stale indicators remain in historical telemetry.
• Auto-relaunch created fresh continuation threads successfully:
  • library continuation: 019c9fee-0556-7173-9509-d6f3b64eaef8 (turn completed)
  • openclaw continuation: 019c9fee-0555-77f0-b351-9ba90c013fd0 (turn entered inProgress then continued)

Updated hypothesis:

• stale spinner appears to be a UI/state reconciliation artifact after stream desync.
• old thread IDs can become unreachable to app-server (thread not found) while UI/log-derived stale state still persists for that thread lineage.
• fallback relaunch is effective for continuity, but root cause remains in stream-state reconciliation / orphaned turn handling.

Requested backend/UI focus:

1. hard reconciliation when source thread is not resolvable but UI still marks streaming.
2. idempotent closure for orphaned inProgress turn markers.
3. stronger handling for thread-stream-state-changed + missing turn references (Turn not found for event).

Sanitized artifact roots used for this validation:

• ~/.codex/reports/codex-stale-thinking/2026-02-27/
• ~/.codex/runtime/relaunched_threads.jsonl

If useful, I can post a compact event timeline matrix keyed by (thread_id, timestamp, event_type) without prompt contents.

[etraut-openai]

I presume this is a bot. Closing.