pipeline.md

The reference pipeline: deep dive

The reference pipeline is an autonomous, multi-agent pipeline for finding memory vulnerabilities in C/C++ codebases. This document explains what each stage of the pipeline does, how to watch a run, and relevant CLI flags.

⚠️ The pipeline spawns autonomous agents and executes target code. The pipeline runs each agent inside a gVisor container with egress restricted to the selected model provider. Agent-spawning subcommands refuse to start outside it unless explicitly overridden. For more information, see security.md and agent-sandbox.md.

This document covers how the reference pipeline works. For the general best practices it implements, see the blog post.

Install and first run

# One-time setup
python3 -m venv .venv && .venv/bin/pip install -e .
./scripts/setup_sandbox.sh   # installs gVisor, builds the agent images, and verifies isolation; note: requires Docker
export ANTHROPIC_API_KEY=sk-ant-...   # or CLAUDE_CODE_OAUTH_TOKEN, or Bedrock — see docs/agent-sandbox.md

# Run the recon → find → verify → report loop
bin/vp-sandboxed run drlibs --model <model-id> --runs 3 --parallel --stream --auto-focus
# Generate a candidate patch for each finding
bin/vp-sandboxed patch results/drlibs/<timestamp>/ --model <model-id>

The default provider is the Anthropic API. Amazon Bedrock and Google Vertex are also supported: pass --provider bedrock or --provider vertex (or set VULN_PIPELINE_PROVIDER) and configure the corresponding credentials before sandbox setup. Provider egress is derived and verified automatically; see agent-sandbox.md.

Start with a small wave like this one to get a feel for how the pipeline works and the token burn before scaling up. Results land in results/<target>/<timestamp>/. With --stream, the first report usually appears within minutes under reports/bug_NN/, so you don't have to wait for the whole batch to finish.

You can drive the pipeline using Claude Code. The repo's CLAUDE.md teaches Claude how to run each phase of the pipeline and what to watch. Launching runs from a Claude Code session makes it easy to tail transcripts, ask what's happening mid-run, and stop early without losing anything.

What each stage does

Build. The target's Dockerfile is built into an ASAN-instrumented image the first time you run a scan against it. The same image is reused for find, grade, and re-attack, so every agent sees the same code in the same environment.

Recon (optional). An agent reads the source tree and proposes a partition of the attack surface ("here are 8 distinct parsers worth attacking separately"). This gives parallel runs different starting places so they don't all converge on the same bug. --auto-focus runs this as a part of the full pipeline. You can skip recon if you've hand-written focus_areas: in the target's config.yaml.

Find. The core part of the loop. Each run gets one agent in its own network-isolated container. The agent reads the source, crafts malformed inputs, and runs the ASAN binary until an input crashes 3 out of 3 times. It outputs the crashing input file (not a written report). Parallel find agents share a found_bugs.jsonl log and must justify why their addition is not a duplicate of something already listed before adding to it.

Grade. A second agent in a fresh container re-runs the PoC and checks that the crash is real (i.e., it reproduces, it's in project code, and it isn't just memory exhaustion). The only thing that crosses from the find container to the grader is the PoC bytes, so the grader isn't influenced by the find agent's reasoning. Flaky-but-real crashes (races, heap-layout-dependent) can pass this step, though they will receive a lower score. Each run's verdict is written to run_NNN/result.json as soon as the grader agent finishes.

Judge. When a finding passes the grader, a short no-tools agent compares the crash against the bugs already in reports/manifest.jsonl and decides whether the finding is a new bug (in which case it's accepted), a cleaner example of a known bug (in which case it replaces the old version), or a duplicate (in which case it's skipped). Judge agents run serially so that two duplicate findings arriving around the same time aren't accidentally both classified as new. The judge stage is only run when the --stream modifier is used.

Report. For each new bug, a report agent writes a structured exploitability analysis using only the PoC and the source. The report includes details on what the corrupted memory lets an attacker do, how reachable it is from real input, a sketch of the escalation path, and a severity. A separate grader agent then scores the report, checking that its claims are backed by evidence (e.g., line numbers, observed re-runs) rather than plausible prose. Reports land in reports/bug_NN/report.json and include the grader's score so you can tell which reports are most trustworthy. The --novelty modifier (off by default) lets the orchestrator check the upstream git history so the report can include whether the bug has already been fixed there. Each completed report also carries a CWE and NIST SP 800-53 Release 5.2.0 mapping. Run bin/vp-sandboxed oscal <results-dir> to aggregate those mappings into an OSCAL 1.1.3 assessment-results document.

Dedup. A separate command that can be run post-hoc to cluster the pipeline results by ASAN signature. It's useful for a quick summary of "these N crashes cluster into M signatures".

Patch. A separate command that generates a candidate patch for each unique bug. For details, see patching.md.

Watching a run

Transcripts and results are written to disk the moment they're produced, so you can watch a run without stopping it:

• Each find and grade agent's transcript lands under results/<target>/<ts>/run_NNN/ as the agent works. Transcripts persist on failed or killed runs.
• found_bugs.jsonl lists every crash submitted so far.
• Each run's result.json is written as soon as the grader finishes reviewing it, so counting run_*/result.json files tells you how many runs are finished.
• Filtering a transcript for "type":"tool_use" shows each command the agent ran. This is the quickest way to see what it's actually doing when you're iterating on prompts.
• With --stream, the reports/ directory also fills in during the run. Judge and report transcripts are saved there, and ls reports/bug_*/report.json shows the reports written so far.

CLI reference

bin/vp-sandboxed recon  <target> --model <m>             # propose focus_areas (prints YAML to stdout)
bin/vp-sandboxed run    <target> --model <m>             # do a single find agent + grade agent run
bin/vp-sandboxed run    <target> --runs N --parallel     # run N find agents at once (spread by focus area)
bin/vp-sandboxed run    <target> --stream                # run judge + report on each crash as its grade lands (recommended)
bin/vp-sandboxed run    <target> --auto-focus            # run recon first and use its partition
bin/vp-sandboxed run    <target> --find-only             # skip grading (useful for prompt iteration)
bin/vp-sandboxed run    <target> --accept-dos            # count DoS-class crashes as valid finds
bin/vp-sandboxed run    <target> --novelty               # reports check upstream git history to determine fix status
bin/vp-sandboxed run    <target> --max-turns N           # per-agent turn budget
bin/vp-sandboxed run    <target> --engagement-context F  # file with your org's authorization scope, threaded into every agent prompt
bin/vp-sandboxed run    <target> --resume <results-dir>  # continue a killed batch, skipping finished runs
bin/vp-sandboxed report results/<target>/<ts>/           # batch-mode reports, for runs done without --stream
bin/vp-sandboxed report results/<target>/<ts>/ --fresh   # redo reports, ignoring existing report.json checkpoints
bin/vp-sandboxed patch  results/<target>/<ts>/           # propose and verify a fix per unique bug
bin/vp-sandboxed dedup  results/<target>/<ts>/           # group crashes by signature
bin/vp-sandboxed oscal  results/<target>/<ts>/           # emit <results-dir>/oscal.json

This reference includes the most commonly used flags. For the full set of flags, use --help on any subcommand.

run, recon, report, and patch accept --provider (or use VULN_PIPELINE_PROVIDER); the model still comes from --model or VULN_PIPELINE_MODEL. --engagement-context is accepted by the agent-spawning commands when you need to supply a written authorization scope.

Design principles

In short, the essential steps of an effective vulnerability-finding pipeline:

1. Build the target.
2. Spin up N agents to search for vulnerabilities.
3. Grade the findings.

We've found it most effective to break this into modular steps, each of which saves its progress durably, and over time we've added more steps like de-duplication, report writing, and so on. A Docker container image is a great way to store a reusable build artifact and provides an environment in which exploits can be attempted safely. Vulnerability-finding agents should store their results in a standard format which can be verified by graders. These agents can decide their own exploration path from the beginning, or you can seed them with "focus areas" via a recon step. Graders should be able to run over any findings multiple times as they're calibrated with human feedback.

Aside from modularity, a critical component is an effective grader. The grader must run as a separate agent with access to a clean sandbox in which it can run any proofs of concept. It should be framed as an adversary actively trying to disprove findings, which are guilty until proven innocent. Proof-of-concept exploits that produce a witness are best, but not always possible. The grader should also be tailored for the vulnerability types under inspection: some bugs are proven by PoC, others by logical argument. Skills or a lightweight routing layer to different verifiers may be good approaches when multiple classes are in scope.

Resume-on-error

Hitting a rate limit or other error mid-run does not lose work. Each agent is one long-lived claude -p session. A 429 or 5xx is first retried with backoff inside the Claude CLI itself. If those retries exhaust, the pipeline runs its own retry loop with backoff. These retries relaunch the agent with the Claude CLI's --resume <session_id>, which restores the full conversation so the agent can continue from the failed turn. This repeats up to 20 times before the run is marked as failed. Even then, you can restart the run using bin/vp-sandboxed run <target> --resume <results-dir>.

We recommend carrying over similar logic if you build your own pipeline.

Usage marker

Outbound API requests from pipeline agents carry a declared usage marker so runbook usage is attributable in Anthropic's request telemetry: an anthropic-cyber-runbook: pipeline header plus a cyber-runbook/<version> leading token in the User-Agent (the pinned claude-cli version stays in the parenthetical). Interactive skill sessions in this repo set only the header — anthropic-cyber-runbook: skills, via .claude/settings.json — and leave the User-Agent untouched.

The marker is structural metadata only: static strings, no request content, no identifiers beyond what the API request already carries. Pipeline agents apply it only when authenticating directly to the Anthropic API (API key or OAuth) — on Bedrock/Vertex the provider rewrites the User-Agent and does not forward custom headers to Anthropic, so pipeline agents send no marker there. The interactive-session header is provider-agnostic; on Bedrock it is SigV4-signed like any other header and stays between you and AWS. It is telemetry, not enforcement — to remove it, set VULN_PIPELINE_NO_TELEMETRY=1 (pipeline) or override ANTHROPIC_CUSTOM_HEADERS in your gitignored .claude/settings.local.json (skills). An ambient ANTHROPIC_CUSTOM_HEADERS in the operator's environment is deliberately not forwarded to the agents — a Claude Code session in this repo injects the skills value into that variable, which would mislabel pipeline traffic.