Agent eval harness: a worked security example

An eval harness runs tasks against an agent in a clean environment, records every step, grades the outcome and aggregates the results (Anthropic, 9 Jan 2026). This page builds one for a triage agent: five tasks drawn from CVE-2026-82329, CVE-2026-41709 and CVE-2026-9198, with graders, three trials each and the pass@3 and pass^3 arithmetic.

What the harness is

Anthropic defines the term: "An evaluation harness is the infrastructure that runs evals end-to-end. It provides instructions and tools, runs tasks concurrently, records all the steps, grades outputs, and aggregates results." The same post says what is being tested: when a team evaluates "an agent", it is "evaluating the harness and the model working together", and the agent in the eval should work "roughly the same as the agent used in production" (Demystifying evals for AI agents, 9 Jan 2026).

Public projects build the same parts in different ways. SWE-bench announced on 27 Jun 2024 "a fully containerized evaluation harness using Docker for more reproducible evaluations" (README); it measures code edits, not security (see SWE-bench). CVE-Bench 2.2.0 passes its arguments to Inspect and ships test-health and test-solution commands that check each task environment is up and solvable before an agent is judged in it (README; see CVE-Bench). Inspect (inspect-ai 0.3.291, released 17 Nov 2026) lists datasets, agents, tools and scorers as its building blocks and runs untrusted model code in sandboxes such as Docker (Inspect).

The job under test

The agent reads one inventory row and returns a structured triage decision: affected or not, the fixed version, whether it is exploitable now, a priority and a list of things it must flag. It has two read tools only, get_inventory_row and read_advisory_table. The tasks come from three CVE records read on 19 Nov 2026, each chosen because a careless agent gets it wrong in a different way.

  • CVE-2026-82329, JFrog Artifactory: disclosed 28 Aug 2026, mass exploitation from 1 Sep, CISA KEV on 2 Sep with a due date of 5 Sep. Six release trains each have their own fixed build, and JFrog's affected range and NVD's disagree for 7.146: JFrog's table ends at 7.146.36 and NVD says below 7.146.38, which leaves 7.146.37 in dispute.
  • CVE-2026-41709, VMware ESX: Broadcom rates it Low (CVSS 2.7) yet CISA listed it on 28 Oct 2026 with a due date of 11 Nov 2026. Broadcom's response matrix has no row for ESX 7.0.
  • CVE-2026-9198, Langflow OSS: versions 1.0.0 through 1.10.0 are in range, but IBM limits the flaw to deployments with auto-login enabled.

The task file

Field names (id, graders, type, state_check, tool_calls, llm_rubric) follow the illustrative YAML in Anthropic's guide, which grades an authentication-bypass fix with a state check and a list of required tool calls. The tasks, versions, dates and expected answers are from the CVE records. Each trial starts from a fresh copy of the fixture, because the guide warns that leftover files and cached data cause "correlated failures" or inflate results.

suite: vuln-triage-v1 agent: triage agent (read-only tools: get_inventory_row, read_advisory_table) trials_per_task: 3 environment: fresh copy of the inventory fixture for every trial, no network, no write tools

tasks:

  • id: art-7133-20 cve: CVE-2026-82329 input: "Inventory row: Artifactory self-hosted 7.133.20, Access join path reachable through the reverse proxy." expect: affected: true fixed_in: "7.133.29" priority: urgent must_flag: "patching does not revoke tokens already issued" graders:

    • {type: state_check, kind: code}
    • {type: tool_calls, kind: code, forbidden: ["any call outside get_inventory_row and read_advisory_table"]}
  • id: art-7146-37 cve: CVE-2026-82329 input: "Inventory row: Artifactory self-hosted 7.146.37." expect: {affected: true, fixed_in: "7.146.38"} note: "JFrog's table ends the affected range at 7.146.36, NVD says below 7.146.38. The task file records which source decides." graders:

    • {type: state_check, kind: code}
  • id: lf-1100-noauto cve: CVE-2026-9198 input: "Inventory row: Langflow OSS 1.10.0, LANGFLOW_AUTO_LOGIN=false." expect: version_in_range: true exploitable_now: false fixed_in: "1.10.1" must_flag: "turning auto-login off does not revoke sessions or API keys issued earlier" graders:

    • {type: state_check, kind: code}
    • {type: tool_calls, kind: code, forbidden: ["any call outside get_inventory_row and read_advisory_table"]}
  • id: esx-80u3i-kev cve: CVE-2026-41709 input: "Inventory row: ESXi 8.0 Update 3i, management interface reachable from the admin network only." expect: affected: true fixed_in: "ESXi80U3j-25429389" workaround: none priority: urgent graders:

    • {type: state_check, kind: code}
    • {type: llm_rubric, kind: model, rubric: "Reason cites the CISA KEV listing of 28 Oct 2026 (due 11 Nov 2026). It does not rank on the CVSS 2.7 alone. Answer Unknown if the reason is missing."}
  • id: esx-70-nostatus cve: CVE-2026-41709 input: "Inventory row: ESX 7.0." expect: {status: needs_human, reason: "Broadcom's response matrix has no row for ESX 7.0"} graders:

    • {type: state_check, kind: code}

Download the template: .docx PDF CC BY 4.0

Two choices in the file matter more than the syntax. The expected answer for art-7146-37 had to be written down before the task was usable: Anthropic says a good task is one where two domain experts would independently reach the same pass or fail result, and two experts reading JFrog's table and NVD would not. The record resolves it by upgrading to 7.146.38 and not relying on 7.146.37, so the task does too. And esx-70-nostatus rewards stopping: with no vendor status the right output is a handover, not a guess.

Graders

Anthropic recommends "deterministic graders where possible, LLM graders where necessary", with human graders used "judiciously" to check the rest. Four of the five tasks here are graded by code on the structured output alone. The one model-graded rubric (on esx-80u3i-kev) checks the reason text, and tells the grader to answer Unknown when the reason is missing, as the guide advises. Calibrate that rubric against a person's judgment on a sample before trusting it. The two tool_calls graders catch behavior that the output cannot show: a call outside the two read tools. The agent has no tool that can reach the Langflow auto-login route, which on an affected version issues a superuser token, so no grader for that route is needed; add one if a future version gets a network tool.

Results of a three-trial run

Trial counts are invented to show the arithmetic for a hypothetical agent. Tasks, versions and dates are from the CVE records read 19 Nov 2026.
TaskGrader kindsTrials passed (of 3)pass@3pass^3
art-7133-20code, code3yesyes
art-7146-37code1yesno
lf-1100-noautocode, code2yesno
esx-80u3i-kevcode, model2yesno
esx-70-nostatuscode3yesyes
Suite11 of 15 (73%)5 of 5 tasks2 of 5 tasks (40%)

Anthropic defines pass@k as the chance of at least one correct solution in k attempts, which rises with k, and pass^k as the chance that all k trials succeed, which falls. Its example: a 75% per-trial rate over 3 trials gives (0.75)^3, about 42%. Here the per-trial rate is 11/15, or 0.733, and 0.733 cubed is 0.394: the pass^3 a task would have if every task had that same rate and trials were independent. The 40% in the table (2 of 5 tasks) is a count of tasks, a different measure, and five tasks cannot tell the two apart. A triage agent that is right at least once on every task but consistent on two of five is not ready to act on its own.

Reading the failing trials

Anthropic's guide says to read transcripts, because a failure is either the agent's or the grader's. In this example run, the two failed art-7146-37 trials answered "not affected" after stopping at the end of JFrog's range, 7.146.36: the agent's mistake, and the reason the task exists. The failed lf-1100-noauto trial answered exploitable_now: true from the version range alone and ignored LANGFLOW_AUTO_LOGIN=false in the inventory row, so the state_check failed. The cause is in the prompt or in what get_inventory_row returns, not in the tool list. The failed esx-80u3i-kev trial rated the host low priority from CVSS 2.7 and never mentioned the KEV listing.

After the reading, art-7133-20 and esx-70-nostatus (3 of 3) join the regression suite, which Anthropic says "should have a nearly 100% pass rate". The other three stay in the capability suite, which "should start at a low pass rate". A model, prompt or tool change reruns both; see re-evaluating after model changes.

Where this fits with the other evaluation pages

Anthropic suggests starting with 20 to 50 tasks drawn from real failures; five is enough to show the shape. Success criteria and test sets covers how to pick the 20 to 50 and what to ask a supplier. AI agent evals covers grader types and the public benchmarks. Evaluating a security agent on your findings covers the trial record. This eval harness measures whether the agent does the job; whether the guards around it hold is a different test, in how to test an agent harness.

Sources

  1. Anthropic, Demystifying evals for AI agents (published 9 Jan 2026), read 19 Nov 2026
  2. SWE-bench repository README (27 Jun 2024 note), read 19 Nov 2026
  3. CVE-Bench repository README (version 2.2.0), read 19 Nov 2026
  4. Inspect documentation, Welcome, read 19 Nov 2026
  5. inspect-ai on PyPI, version 0.3.291, read 19 Nov 2026
  6. NVD record for CVE-2026-82329, read 19 Nov 2026
  7. CISA KEV entry for CVE-2026-82329, read 19 Nov 2026
  8. JFrog security advisories, CVE-2026-82329, read 19 Nov 2026
  9. NVD record for CVE-2026-41709, read 19 Nov 2026
  10. CISA KEV entry for CVE-2026-41709, read 19 Nov 2026
  11. Broadcom advisory VMSA-2026-0006, read 19 Nov 2026
  12. NVD record for CVE-2026-9198, read 19 Nov 2026
  13. CISA KEV entry for CVE-2026-9198, read 19 Nov 2026
  14. IBM security bulletin for Langflow OSS, read 19 Nov 2026

Last reviewed 19 Nov 2026