Judge agent request outcome (completed/failed) via LLM instead of tool-failure counting#19185
Merged
jvcalderon merged 5 commits intoJul 14, 2026
Merged
Conversation
resolveFinalRequestOutcome's needs_input short-circuit also fired in the escalation resume path, where the request is always in needs_input at that point - so an approved escalation never actually resolved the request. Add isHumanResponse so a resume can judge through needs_input, mirroring updateRequestStatus. Also fixes two semantics: a human rejecting an escalation is no longer forced to completed mechanically, it now goes through the same LLM judgment; and resolving one escalation no longer finalizes a request that still has other escalations pending a human response.
jvcalderon
changed the title
jvcalderon
marked this pull request as ready for review
July 9, 2026 15:56
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Confidence score: 5/5
- Safe to merge after the addressed issues were fixed.
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…RequestOutcome The try/catch also wrapped the initial tryGet and pending-escalation lookup, so a failure in either fell back to a mechanical completed/failed status without knowing whether the request was still waiting on a human. Narrow the fallback to generateRequestOutcome failures only, and let guard-query failures propagate.
adrinr
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Until now, whether a tracked agent request ended up
completedorfailedwas decided mechanically, purely by counting tool call failures (resolveFinalRequestStatus). That criterion is wrong in both directions: a tool call can fail and still have the user's actual goal met some other way (falsefailed), and every tool call can technically succeed without ever delivering what the user asked for (falsecompleted).The core change in this PR is a new function,
generateRequestOutcome(agentRequests/helpers.ts), that asks the LLM to judge - once per request finalization, not per tool call - whether the user's underlying goal was actually achieved, based on the request's full action timeline (user messages, tool calls and their outcomes) plus the agent's final reply.resolveFinalRequestOutcome(agentRequests/crud.ts) wires this judgment into the three places a request can finish (chat stream, webhook chat, escalation resume), falling back to the old mechanical criteria only if the LLM call itself fails.On top of that core change, this PR also fixes issues found while wiring the judge into the escalation resume path:
needs_inputshort-circuit also fired during escalation resume, where the request is always inneeds_inputat that point - so an approved escalation never actually resolved the request, leaving it stuck. Fixed with anisHumanResponseflag (mirroring the one already onupdateRequestStatus).completed- it now goes through the same LLM judgment as any other finalization, since a rejection is a legitimate outcome but not automatically a success or a failure.Screen recording
Judge as completed
Screen.Recording.2026-07-09.at.17.48.47.mov
Judge as failed
Screen.Recording.2026-07-09.at.17.51.04.mov
Launchcontrol
Agent requests are now marked completed or failed based on whether the user's actual goal was achieved, judged by the AI itself from the full record of what happened - not just by whether every tool call technically succeeded.