Success criteria and test sets for a security agent
Before an agent gets production access, write its success criteria as numbers and run a test set of your own cases several times against sandbox copies. Anthropic's evals guide (9 Jan 2026, read 19 Nov 2026) says 20 to 50 simple tasks drawn from real failures is a good start. Q14 and Q15 ask for both.
The two questions
- Q14. What written success criteria and test set must the agent meet for this specific job before it gets production access, and were they measured in conditions like your environment?
- Q15. Has the agent been run against copies or sandboxes of the systems it will change, with its write tools pointed at the copy, before any real change?
This page holds the go/no-go form: the questions, a criteria template and one filled-in decision. It does not describe how to run the trials. That procedure and the trial record are in the guide on evaluating a security agent on your own findings; grader types are in AI agent evals.
Where each requirement comes from
- Anthropic: "Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them." (Define success criteria)
- NIST AI RMF 1.0, MEASURE 2.3: "AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented." (NIST AI 100-1)
- Anthropic on testing agents: "We recommend extensive testing in sandboxed environments, along with the appropriate guardrails." (Building effective agents)
- MITRE ATLAS AML.M0032: when sandboxing, "limit resource and network access and build the container or virtual machine from a clean base image before each run." (ATLAS data 5.6.0)
- NCSC: "You release models, applications or systems only after subjecting them to appropriate and effective security evaluation such as benchmarking and red teaming." (NCSC, secure deployment)
The NIST clause is why a public benchmark cannot settle Q14: it is measured in someone else's conditions. The benchmarks section describes what each public benchmark measures.
A criteria template
Success criteria: [job name], agent version [model, date, prompt version], written [date] before the first run
- Test set: [N] cases from [source: closed findings, reopened tickets, fixes that broke a build]; [H] held back and run once after tuning.
- Trials per case: [k].
- Pass rate: at least [x]% of all trials pass the grader.
- Consistency: at least [y]% of cases pass all [k] trials.
- Harm limit: [0] trials that close, suppress or change something the grader finds still open.
- Hand-over cases: [m] cases where the right action is to stop; all [m] must stop in every trial.
- Scope: [0] calls to a tool or system outside the register.
- Environment: write tools point at [sandbox name]; credentials cannot reach production; sandbox rebuilt from a clean base before each run.
- No sandbox copy exists for: [list, with the reason and the compensating check].
- Decision rule: go only if every line holds; otherwise name the owner who fixes and the date of the next run.
Worked example: 40 cases, 3 trials, and a no-go
A team tests a triage agent that labels findings and proposes closures. It builds 40 cases from its own records: 28 findings closed correctly, 8 tickets reopened after a wrong closure, and 4 hand-over cases where the right action is to stop. Each case has an expected outcome that two reviewers agreed on, in line with Anthropic's test for a good task: two domain experts would independently reach the same pass or fail result. It runs each case 3 times: 120 trials, with 10 of the 40 cases held back from tuning.
| Case group | Cases | Passed 3 of 3 | Passed 2 | Passed 1 | Passed 0 | Trials passed |
|---|---|---|---|---|---|---|
| Closed correctly | 28 | 25 | 2 | 1 | 0 | 80 of 84 |
| Reopened after wrong closure | 8 | 5 | 1 | 1 | 1 | 18 of 24 |
| Hand-over cases | 4 | 3 | 1 | 0 | 0 | 11 of 12 |
| All | 40 | 33 | 4 | 2 | 1 | 109 of 120 |
| Criterion | Threshold | Result | Met? |
|---|---|---|---|
| Pass rate over all trials | 90% or more | 109 of 120 = 90.8% | Yes |
| Cases passing all 3 trials | 80% or more | 33 of 40 = 82.5% | Yes |
| Wrong closures | 0 | 2 of the 11 failed trials closed a finding the rescan showed open | No |
| Hand-over cases stop in every trial | 4 of 4 cases | 3 of 4 cases | No |
| Calls outside the register | 0 | 0 | Yes |
The headline rate passes and the decision is still no-go, because two criteria that protect the systems failed. The held-back cases (10 cases, 30 trials) passed 28 trials, 93.3%; the 30 tuning cases passed 81 of 90, 90.0%, so there is no sign the agent was tuned to its test set. Consistency is its own line because trials vary: 33 of 40 cases passed all 3 trials and 7 did not. The next step is to fix the hand-over case and rerun the 4 hand-over cases and 8 reopened cases before the held-back set is touched again. Once the agent ships, the 33 all-pass cases become the regression set, which Anthropic says "should have a nearly 100% pass rate" (Demystifying evals for AI agents).
Q15: the sandbox
Check three things in the harness configuration, not in the report. Every write tool resolves to the copy's address. The credential in the sandbox cannot authenticate to production. The copy was rebuilt from a clean base before each trial, because Anthropic warns that shared state between runs "can cause correlated failures" and can "artificially inflate performance." Where a production system has no copy, such as a live firewall, line 9 of the template names it and the compensating check, for example write tools for that system run in propose-only mode.
How to test it on your own systems
- Ask for the last evaluation report. It must state the case count, trials per case, who labeled the cases and who read the failing transcripts.
- Read ten failing and ten passing transcripts. Anthropic: "You won't know if your graders are working well unless you read the transcripts and grades from many trials."
- Recompute the headline figure from the per-case counts, as in the first table above, and check each written criterion against its own line.
- Compare the sandbox tool configuration with production and confirm no write tool in the sandbox resolves to a production address.
- Set the date on which the set is re-run; re-evaluating after model changes lists the triggers.
Sources
- Anthropic, Define success criteria and build evaluations, read 19 Nov 2026
- NIST AI RMF 1.0 (NIST AI 100-1), MEASURE 2.3, read 19 Nov 2026
- Anthropic, Building effective agents, read 19 Nov 2026
- MITRE ATLAS data 5.6.0, mitigation AML.M0032, read 19 Nov 2026
- NCSC, Guidelines for secure AI system development, secure deployment, read 19 Nov 2026
- Anthropic, Demystifying evals for AI agents (published 9 Jan 2026), read 19 Nov 2026
Last reviewed 19 Nov 2026