AI agent evals: grading a run and setting the release gate

Grade an agent eval by running every task several times and reporting pass@1, pass@3 and pass^3. In the worked run below (assumed figures, 40 findings, 3 trials each), pass@1 is 87.5 percent and pass^3 is 77.5 percent. Against a gate of 80 written before the run, the release is blocked.

Three numbers from one set of trials

Anthropic's post on agent evals explains why one run is not enough: "a task that passed on one eval run might fail on the next." It defines two metrics. "pass@k measures the likelihood that an agent gets at least one correct solution in k attempts" and rises as k rises. "pass^k measures the probability that all k trials succeed" and falls as k rises. Its example: a 75 percent per-trial success rate over 3 trials gives (0.75)³, about 42 percent, for passing all three (Demystifying evals for AI agents).

For a security agent that acts on findings, pass^k is the number to gate on: a triage agent that is right on the second try is wrong on the first, and the first answer is the one that closes the ticket. pass@k shows what the agent can do given several tries. Building the set of findings is covered in evaluate a security agent on your own findings; the parts of an eval harness are on the evaluation harness page. This page covers what happens after the trials finish: grading them and deciding whether the agent ships.

One task, graded by code

The agent's job in this example is to decide, for a dependency finding, whether the host is affected and which version fixes it. The task family comes from CVE-2026-75604, a Next.js path traversal in the incremental cache (GitHub advisory GHSA-p293-qw3h-jr36, published 25 Aug 2026). The advisory lists affected versions as next 13.4 up to but excluding 15.5.24, and 16.0 up to but excluding 16.3.3, with patched versions 15.5.24 and 16.3.3. It describes the flaw for applications using the Pages router and the App router without Cache Components "when the server is hosted on machines using a Windows filesystem", and says there is no known workaround for affected Windows-hosted applications. It names no other host type, so the labels below treat them as not affected on the advisory's stated condition.

A human writes the right answer for each task before the run. A code grader then compares two fields of the agent's output with the labeled ones, so no model is needed to judge it.

Four variants of one finding; the version ranges and fixed versions are the advisory's, read 19 Nov 2026. The labels follow the advisory's stated conditions.
Task inputLabeled answerCode grader checks
next 15.5.20, Windows Server host, Pages RouterAffected; fixed in 15.5.24Decision field equals "affected"; fix version is in the advisory's fixed list
next 16.3.2, Windows Server host, Pages RouterAffected; fixed in 16.3.3Same two checks
next 15.5.20, Linux container, Pages RouterNot affected on this host (no Windows filesystem)Decision field equals "not affected"
next 16.3.3, Windows Server hostNot affected (already fixed)Decision field equals "not affected"

The pass or fail on the decision field stays in code. A model grader is used only for the written reason the agent gives, and only after the calibration below.

A worked run: 40 tasks, 3 trials, 120 results

The figures below are assumed to show the arithmetic; they are not measurements of any product. The held-back set has 40 tasks like those above, each run 3 times, giving 120 trials. The grader recorded how many of the 3 trials passed on each task.

Assumed results of one run. 31 x 3 + 5 x 2 + 2 x 1 = 105 passing trials of 120.
  • 3Tasks31Passing trials93
  • 2Tasks5Passing trials10
  • 1Tasks2Passing trials2
  • 0Tasks2Passing trials0
  • TotalTasks40Passing trials105
  • pass@1 = 105 / 120 = 87.5 percent. This is the average single-trial success rate.
  • pass@3 = tasks with at least one passing trial = 31 + 5 + 2 = 38 of 40 = 95 percent.
  • pass^3 = tasks where all three trials passed = 31 of 40 = 77.5 percent.
  • Anthropic's shortcut would predict (0.875)³ = 67 percent. The run is higher because tasks are not equal: 31 are reliable and 9 cause all the trouble. Use the counted figure, not the shortcut.

Split the 15 failing trials by error type, because the two errors cost different amounts. Assume 6 were a wrong "not affected" (a false clear that leaves a vulnerable host open) and 9 were a wrong "affected" (extra work for an owner). A false clear is the error to gate on separately.

Calibrating a model grader, in numbers

Anthropic lists model-based graders as flexible but "non-deterministic" and as requiring "calibration with human graders for accuracy". The figures here are assumed. A model grader reads the agent's written reason for each of 120 trials and marks it sound or unsound. Before it is used, a reviewer grades a random sample of 30 of those reasons without seeing the model's mark. The rule is written first: the model grader is accepted only if it agrees with the reviewer on at least 27 of 30 (90 percent) and gives at most 1 sound mark to a reason the reviewer finds unsound.

Assumed calibration sample of 30 reasons. Agreement is (18 + 7) / 30 = 25 of 30, 83.3 percent.
  • Model: soundReviewer: sound18Reviewer: unsound3Total21
  • Model: unsoundReviewer: sound2Reviewer: unsound7Total9
  • TotalReviewer: sound20Reviewer: unsound10Total30

Agreement is 25 of 30, below 27, and the 3 sound marks on unsound reasons are the costly kind: they let a lucky guess pass. The model grader is not accepted. Reasons stay with the reviewer, the rubric is rewritten (for example, "names the Windows filesystem condition and the fixed version"), and a new sample of 30 different reasons is graded. Reusing the same 30 would measure the rewritten rubric against the cases that shaped it. The cost of acceptance is the reviewer's time on 30 reasons, a one-off per rubric; it buys a grader for the other 90.

Write the gate before the run

The gate in this example, written down on the day the set was frozen: pass^3 of at least 80 percent on the held-back set, and no false clear on any task whose CVE is in the CISA KEV catalog. CVE-2026-75604 is not in the KEV catalog (catalog version 2026.11.18), so the second condition bites on other tasks in the set; assume 2 of the 6 false-clear trials fall on KEV-listed tasks. The run above gets 77.5 percent, so both conditions fail and the release is blocked.

A failed gate is not argued away. The nine tasks with a failed trial have now been examined, so they move to the tuning pile: a gate that reuses them measures memory. Fix the cause in the prompt or tools, then rerun on a held-back set that excludes them.

Record the gate

The record is one page, kept with the agent's version table. It names the role that wrote the gate, the role that signed the decision, and the dates, so a later reader can see the gate was fixed before the result. Roles, set name and run ID below are constructed.

Release gate record: patch-triage agent, dependency findings Agent version: 2 Nov 2026.1 (prompt, tool list and model ID string frozen) Held-back set: HB-40, 40 tasks, frozen 2 Nov 2026 Trials per task: 3 Gate, written 2 Nov 2026 by the evaluation owner, before any trial:

  1. pass^3 on HB-40 of at least 80 percent
  2. no false-clear trial on any task whose CVE is in the CISA KEV catalog Run: 9 Nov 2026, run ID ev-20261109-01 Result: pass@1 87.5 percent, pass@3 95 percent, pass^3 77.5 percent (31 of 40); 6 false-clear trials, 2 on KEV-listed tasks Model grader for reasons: not accepted (25 of 30 agreement, 3 false sound marks); reasons graded by a reviewer Decision: BLOCKED. Condition 1 missed by 2.5 points; condition 2 failed Decision signed by: release owner, 9 Nov 2026 Next: examine the 9 unstable tasks, move them to the tuning set, rerun on a held-back set that excludes them

Download the template: .docx PDF CC BY 4.0

Two kinds of suite, two kinds of gate

Anthropic separates capability evals, which "should start at a low pass rate" and give the team "a hill to climb", from regression evals, which "should have a nearly 100% pass rate" and protect against backsliding. The 80 percent gate above suits a first release of a new agent. Once tasks pass reliably they join a regression suite whose gate is near 100 percent, run before every change of model, prompt or tool list (see re-evaluate after model changes).

Public benchmarks measure models on public code, so none can stand in for either suite. The benchmarks index lists what each measures; use them to choose a model to try, then run your own set.

Sources

  1. Anthropic, Demystifying evals for AI agents, read 19 Nov 2026
  2. Vercel, Next.js advisory GHSA-p293-qw3h-jr36 (CVE-2026-75604), read 19 Nov 2026
  3. CISA, Known Exploited Vulnerabilities catalog, read 19 Nov 2026

Last reviewed 19 Nov 2026