Evaluate a security agent on your own findings
Run the agent on about 40 of your own closed findings, check each result in a clean copy of the system, and decide widen, hold or stop against thresholds written before the run. In Snyk's VulnBump (15 Oct 2026), an unaided agent claimed fixes that a rescan still found in 70 of its 124 failures.
Check the result in a clean copy, not in the transcript
Anthropic's eval guide separates the transcript from the outcome: a flight-booking agent may write "Your flight has been booked", but "the outcome is whether a reservation exists in the environment's SQL database". For a security fix the outcome has four parts. Snyk's VulnBump benchmark copied each agent's changes into a clean container and counted a pass only if all four held:
- The dependencies install with the command CI would run.
- A new scan confirms the vulnerabilities are gone.
- The project builds.
- Every test that passed before still passes.
Snyk ran 307 real upgrades (2,066 fixable vulnerabilities, 43 repositories, six package managers) with all agents on Claude Sonnet 5.5 and a limit of 110 turns. These are Snyk's own figures, listed alphabetically, and Snyk's Evo Remediation Agent is its own product:
| Configuration | Upgrades passed | What it was given |
|---|---|---|
| Claude Code with Snyk scan | 77.5% | Snyk's recommended target version and a Snyk scan |
| Claude Code unaided | 59.6% | The task only; it had to find the safe version itself |
| Evo Remediation Agent | 90.2% | Snyk's agent, run in auto-approve mode |
| Package manager bump (no AI) | 46.9% | Snyk's recommended version applied, nothing else |
Keeping the build green was not what separated them: all three agents did that about 93% of the time. Removing the vulnerability was. In 70 of its 124 failures, Claude Code unaided said it had fixed vulnerabilities that a rescan still found. A transcript-based check would have counted those 70 as successes.
How many findings are enough
Anthropic's guide says "20-50 simple tasks drawn from real failures is a great start" and that more mature agents may need larger sets to detect smaller effects. The table shows why, using 95% Wilson intervals computed by this site from Snyk's counts and from 40 findings at the same rates.
| Result | Passes | Interval for the true pass rate |
|---|---|---|
| Claude Code unaided, 307 upgrades | 183 of 307 (59.6%) | 54.0% to 64.9% |
| Package manager bump, 307 upgrades | 144 of 307 (46.9%) | 41.4% to 52.5% |
| Same 59.6% rate on 40 findings | 24 of 40 | 44.6% to 73.7% |
| Same 46.9% rate on 40 findings | 19 of 40 | 32.9% to 62.5% |
On 307 upgrades the two intervals do not overlap. On 40 findings they overlap by about 18 points. A set of 40 finds large failures, such as a fifth of confident claims being wrong, but cannot tell two agents apart that differ by 12 points. Pick the thresholds before the run with that width in mind.
A worked trial on 40 findings
This trial is an invented example to show the record filled in. Job: an agent opens a draft pull request that upgrades one vulnerable dependency. Set: 40 findings from the backlog, drawn from every ecosystem the team runs, labeled on 2 Nov 2026 by two reviewers who agreed a reference fix or a reason not to fix for each. The prompt was tuned on a different set of 10; these 40 were never used while tuning. Thresholds written on 2 Nov: at least 75% of the 40 verified, and at most 5% of the 40 claimed fixed but failing verification.
| Step | Count | Note |
|---|---|---|
| Findings in the set | 40 | |
| Agent opened a pull request (claimed a fix) | 34 | |
| Agent stopped under its own stop conditions | 6 | Reference answer: 4 were fixable (missed), 2 correctly declined |
| Claimed fixes: dependencies install | 33 of 34 | |
| Claimed fixes: rescan clear | 29 of 34 | |
| Claimed fixes: build passes | 31 of 34 | |
| Claimed fixes: prior tests still pass | 30 of 34 | |
| Passed all four checks | 27 of 40 (67.5%) | 95% interval 52.0% to 79.9% |
| Claimed fixed but failed a check | 7 of 40 (17.5%) | Wrong and confident |
| Reviewer time | 423 minutes | 27 verified x 11 minutes, 7 failed x 18 minutes |
| Token cost | $18.00 | 40 x $0.45, the Task B figure on the cost page |
Decision, recorded with the name of the person who made it: hold. The verified rate of 67.5% is under the 75% threshold, and the lower end of its interval is 52.0%. The 17.5% of wrong, confident claims is more than three times the 5% limit. The agent stays at advisory level, the five rescan failures are read in full, and the stop conditions are changed before a second run on a new set. Cost per verified fix is 15.7 minutes of review ($0.67 in tokens), a figure to compare with the cost page and the build vs buy guide.
How trials fail
- The check trusts the agent's summary. VulnBump's 70 false fix claims are the case.
- Trials share state. Anthropic reports Claude "examining the git history from previous trials" and gaining an unfair advantage; start every trial from a fresh copy and point write tools at copies only.
- Labels are ambiguous. Anthropic: a good task is one where two domain experts would independently reach the same pass or fail result.
- The set is one-sided. Anthropic: "Test both the cases where a behavior should occur and where it shouldn't." Include findings where the right answer is to decline.
- The set was used for tuning. Keep a held-back pile and run it once.
- Blocked runs vanish. Record a block as its own outcome, as described on the model choice page.
- Nobody read the failures. Anthropic: read the transcripts of failed trials to learn whether the agent erred or your grader rejected a valid fix.
The trial record
Trial record
- Job and agent version: model ID and date, effort setting, prompt version, tool list
- Test set: number of findings, where drawn from, who labeled them, date labeled, which findings are held back
- Thresholds written before the run (copy them here with the date)
- Verification method: the checks run in a clean copy and who ran them
- Per finding: agent decision, reference answer, each check passed or failed
- Wrong-but-confident cases listed in full
- Blocked runs and retries, counted separately
- Reviewer minutes per accepted change and per rejected one
- Tokens used and cost per finding
- Stop conditions that fired; any change attempted outside scope
- Interval for the main rate (state the method)
- Decision: widen, hold or stop, with the name of the person who decided and the date
The trial stops being valid when something changes. The UK NCSC's secure AI guidelines say update processes should reflect that "changes to data, models or prompts can lead to changes in system behaviour (for example, you treat major updates like new versions)". A new model, prompt or tool means a new trial or at least a rerun of the held-back set; the re-evaluation page lists the triggers. For grader types and public benchmarks see AI agent evals, and for the checklist questions the success criteria and test sets page.
Sources
- Snyk, VulnBump (15 October 2026), read 19 Nov 2026
- Anthropic, Demystifying evals for AI agents, read 19 Nov 2026
- UK NCSC, Guidelines for secure AI system development, secure operation and maintenance, read 19 Nov 2026
Last reviewed 19 Nov 2026