Evaluate a security agent on your own findings

Run the agent on about 40 of your own closed findings, check each result in a clean copy of the system, and decide widen, hold or stop against thresholds written before the run. In Snyk's VulnBump (15 Oct 2026), an unaided agent claimed fixes that a rescan still found in 70 of its 124 failures.

Check the result in a clean copy, not in the transcript

Anthropic's eval guide separates the transcript from the outcome: a flight-booking agent may write "Your flight has been booked", but "the outcome is whether a reservation exists in the environment's SQL database". For a security fix the outcome has four parts. Snyk's VulnBump benchmark copied each agent's changes into a clean container and counted a pass only if all four held:

  1. The dependencies install with the command CI would run.
  2. A new scan confirms the vulnerabilities are gone.
  3. The project builds.
  4. Every test that passed before still passes.

Snyk ran 307 real upgrades (2,066 fixable vulnerabilities, 43 repositories, six package managers) with all agents on Claude Sonnet 5.5 and a limit of 110 turns. These are Snyk's own figures, listed alphabetically, and Snyk's Evo Remediation Agent is its own product:

Snyk, VulnBump, 15 Oct 2026, read 19 Nov 2026. Snyk states it tested one model, ran each upgrade once, and used Snyk Open Source as the judge.
ConfigurationUpgrades passedWhat it was given
Claude Code with Snyk scan77.5%Snyk's recommended target version and a Snyk scan
Claude Code unaided59.6%The task only; it had to find the safe version itself
Evo Remediation Agent90.2%Snyk's agent, run in auto-approve mode
Package manager bump (no AI)46.9%Snyk's recommended version applied, nothing else

Keeping the build green was not what separated them: all three agents did that about 93% of the time. Removing the vulnerability was. In 70 of its 124 failures, Claude Code unaided said it had fixed vulnerabilities that a rescan still found. A transcript-based check would have counted those 70 as successes.

How many findings are enough

Anthropic's guide says "20-50 simple tasks drawn from real failures is a great start" and that more mature agents may need larger sets to detect smaller effects. The table shows why, using 95% Wilson intervals computed by this site from Snyk's counts and from 40 findings at the same rates.

Wilson 95% intervals calculated on 19 Nov 2026 from the counts shown. The 40-finding rows are the same rates scaled down, not Snyk results.
ResultPassesInterval for the true pass rate
Claude Code unaided, 307 upgrades183 of 307 (59.6%)54.0% to 64.9%
Package manager bump, 307 upgrades144 of 307 (46.9%)41.4% to 52.5%
Same 59.6% rate on 40 findings24 of 4044.6% to 73.7%
Same 46.9% rate on 40 findings19 of 4032.9% to 62.5%

On 307 upgrades the two intervals do not overlap. On 40 findings they overlap by about 18 points. A set of 40 finds large failures, such as a fifth of confident claims being wrong, but cannot tell two agents apart that differ by 12 points. Pick the thresholds before the run with that width in mind.

A worked trial on 40 findings

This trial is an invented example to show the record filled in. Job: an agent opens a draft pull request that upgrades one vulnerable dependency. Set: 40 findings from the backlog, drawn from every ecosystem the team runs, labeled on 2 Nov 2026 by two reviewers who agreed a reference fix or a reason not to fix for each. The prompt was tuned on a different set of 10; these 40 were never used while tuning. Thresholds written on 2 Nov: at least 75% of the 40 verified, and at most 5% of the 40 claimed fixed but failing verification.

Invented results for illustration.
StepCountNote
Findings in the set40
Agent opened a pull request (claimed a fix)34
Agent stopped under its own stop conditions6Reference answer: 4 were fixable (missed), 2 correctly declined
Claimed fixes: dependencies install33 of 34
Claimed fixes: rescan clear29 of 34
Claimed fixes: build passes31 of 34
Claimed fixes: prior tests still pass30 of 34
Passed all four checks27 of 40 (67.5%)95% interval 52.0% to 79.9%
Claimed fixed but failed a check7 of 40 (17.5%)Wrong and confident
Reviewer time423 minutes27 verified x 11 minutes, 7 failed x 18 minutes
Token cost$18.0040 x $0.45, the Task B figure on the cost page

Decision, recorded with the name of the person who made it: hold. The verified rate of 67.5% is under the 75% threshold, and the lower end of its interval is 52.0%. The 17.5% of wrong, confident claims is more than three times the 5% limit. The agent stays at advisory level, the five rescan failures are read in full, and the stop conditions are changed before a second run on a new set. Cost per verified fix is 15.7 minutes of review ($0.67 in tokens), a figure to compare with the cost page and the build vs buy guide.

How trials fail

  • The check trusts the agent's summary. VulnBump's 70 false fix claims are the case.
  • Trials share state. Anthropic reports Claude "examining the git history from previous trials" and gaining an unfair advantage; start every trial from a fresh copy and point write tools at copies only.
  • Labels are ambiguous. Anthropic: a good task is one where two domain experts would independently reach the same pass or fail result.
  • The set is one-sided. Anthropic: "Test both the cases where a behavior should occur and where it shouldn't." Include findings where the right answer is to decline.
  • The set was used for tuning. Keep a held-back pile and run it once.
  • Blocked runs vanish. Record a block as its own outcome, as described on the model choice page.
  • Nobody read the failures. Anthropic: read the transcripts of failed trials to learn whether the agent erred or your grader rejected a valid fix.

The trial record

Trial record

  • Job and agent version: model ID and date, effort setting, prompt version, tool list
  • Test set: number of findings, where drawn from, who labeled them, date labeled, which findings are held back
  • Thresholds written before the run (copy them here with the date)
  • Verification method: the checks run in a clean copy and who ran them
  • Per finding: agent decision, reference answer, each check passed or failed
  • Wrong-but-confident cases listed in full
  • Blocked runs and retries, counted separately
  • Reviewer minutes per accepted change and per rejected one
  • Tokens used and cost per finding
  • Stop conditions that fired; any change attempted outside scope
  • Interval for the main rate (state the method)
  • Decision: widen, hold or stop, with the name of the person who decided and the date

Download the template: .docx PDF CC BY 4.0

The trial stops being valid when something changes. The UK NCSC's secure AI guidelines say update processes should reflect that "changes to data, models or prompts can lead to changes in system behaviour (for example, you treat major updates like new versions)". A new model, prompt or tool means a new trial or at least a rerun of the held-back set; the re-evaluation page lists the triggers. For grader types and public benchmarks see AI agent evals, and for the checklist questions the success criteria and test sets page.

Sources

  1. Snyk, VulnBump (15 October 2026), read 19 Nov 2026
  2. Anthropic, Demystifying evals for AI agents, read 19 Nov 2026
  3. UK NCSC, Guidelines for secure AI system development, secure operation and maintenance, read 19 Nov 2026

Last reviewed 19 Nov 2026