# Security AI Agent Benchmarks: Results and Method Table

Six benchmark families, in eight rows listed alphabetically, show what each measures, how it grades and its latest published figures with model versions and dates, read 19 Nov 2026. The figures answer different tasks and cannot be compared across rows. None tests an agent on your own findings.

## Every benchmark in one table

Rows are alphabetical, which implies no preference. "Latest" means the newest figures in the paper or on the project's leaderboard on 19 Nov 2026; a figure applies to the version, setup and number of attempts stated beside it.

Sources: each benchmark's paper or Meta post, the CyberGym leaderboard data file at cybergym.io, the SWE-bench leaderboards data at swebench.com, and the CVE-Bench and BountyBench repositories, all read 19 Nov 2026. BountyBench's repository README lists no leaderboard, so its paper is the published result.

| Benchmark | What it measures | How it grades | Latest published results | Page |
|---|---|---|---|---|
| AutoPatchBench (part of CyberSecEval 4) | Repair of 136 fuzzing-found C/C++ vulnerabilities (a 113-sample Lite subset was used in the case study) | Patch must build and stop the crash, then pass 10 minutes of fuzzing and white-box differential testing | Meta post, 29 Apr 2025, 113-sample Lite set: Gemini 1.5 Pro 61.1% of samples got a patch that built and stopped the crash, 5.3% of samples got a correct patch after verification; all models about 60% and 5% to 11% | No page |
| BountyBench | Detect, Exploit and Patch on 25 open-source systems: 40 bug bounties, 120 tasks | Exploit verifier on current and patched code; a patch passes only if invariant tests pass and the exploit fails | Paper v3, 2 Dec 2025, up to 3 attempts: Patch 90.0% (Codex CLI with o3-2025-04-16 and with o4-mini-2025-04-16), Exploit 67.5% (claude-3-7-sonnet-20250219 thinking), Detect 12.5% (Codex CLI, o3-2025-04-16) | [BountyBench](/benchmarks/bountybench/) |
| CTIBench | Threat intelligence knowledge: 2,500 multiple-choice questions, CWE mapping, CVSS vector prediction, ATT&CK extraction, actor attribution | Accuracy against reference answers; CVSS by mean absolute deviation (lower is better); ATT&CK by macro-F1 | Paper v3, 11 Nov 2024, temperature 0: ChatGPT-4 (gpt-4-turbo) 71.0% multiple choice, 72.0% CWE mapping, macro-F1 0.6388 on ATT&CK; CVSS deviation 1.31 for ChatGPT-4 and 1.09 for Gemini-1.5 | [CTIBench](/benchmarks/ctibench/) |
| CVE-Bench | Exploiting 40 critical-severity web application CVEs in a sandbox, zero-day and one-day settings | Automatic grader detects any of eight attack outcomes | Paper v4, 24 Jun 2025, gpt-4o-2024-11-20, five repeats: up to 10% zero-day and 12.5% one-day (13% in the abstract). Repository README shows version 2.2.0 | [CVE-Bench](/benchmarks/cve-bench/) |
| CyberGym | Reproducing 1,507 OSS-Fuzz vulnerabilities in 188 projects from a description and the pre-patch code (four information levels) | Generated test crashes the pre-patch build under sanitizers but not the patched build | Paper v3, 24 Mar 2026, level 1: OpenHands with Claude-Sonnet-4 17.9% on the full set; GPT-5 with high reasoning 22.0% on a 300-instance subset. Project leaderboard, 19 Nov 2026: 95 self-submitted level 1 entries from 15 May 2025 to 24 Oct 2026; Claude Mythos Preview 83.1% (7 Apr 2026), GPT-5.5 81.8% (23 Apr 2026), highest entry 99.38% (DeepSeek-V4.1-Flash, 23 Sep 2026) | [CyberGym](/benchmarks/cybergym/) |
| CyberSecEval (versions 1 and 3) | Insecure code suggestions, compliance with cyberattack requests; version 3 adds spear-phishing, autonomous offense and prompt injection | 189 static analysis rules for 50 insecure practices; LLM judges for the rest | Version 1 paper, 7 Dec 2023, seven Llama 2, Code Llama and GPT models: vulnerable code in 30% of cases, 53% compliance with attack requests (averages). Version 3, 2 Aug 2024: insecure autocomplete failure 31% across models; prompt injection 22% for Llama 3 405B | No page |
| CyberSOCEval (part of CyberSecEval 4) | Malware analysis (609 questions) and threat intelligence reasoning (588 questions) for SOC work | Multiple choice: correct only if all right options and no others are chosen | Paper v2, 10 Nov 2025, seven models (claude-3-7-sonnet, deepseek_r1, gpt-o3, llama4-maverick, gemini-2.5-pro, gpt-4o, llama4-scout): 23.0% to 33.5% on malware analysis, 43.1% to 52.9% on threat intelligence from page images (six models), against random-guess baselines of 0.63% and 1.7% | No page |
| SWE-bench | 2,294 GitHub issues from 12 Python repositories; variants Lite (300), Verified (500), Multilingual (300) | Tests that failed before the fix pass, and tests that passed still pass | swebench.com leaderboards, 19 Nov 2026: Verified, 180 entries, highest resolved 79.2% (Sonar Foundation Agent with claude-opus-4-5, 5 Dec 2025; live-SWE-agent with claude-opus-4-5-20251101, 15 Dec 2025); full 2,294-issue set 52.62% (Sonar Foundation Agent, 19 Dec 2025). First paper baseline, Oct 2023: 1.96% (Claude 2) | [SWE-bench](/benchmarks/swe-bench/) |

## Why figures from different rows do not compare

Four things change what a percentage means, each shown by a pair of numbers from the table's sources.

- Attempts. On the CyberGym leaderboard, the Anthropic Agent entries dated 29 Sep 2025 list Claude Sonnet 4.5 at 28.9% with 1 trial and 66.7% with 30 trials. The page says a trial count is attempts per instance and an instance counts as solved if any trial succeeds. BountyBench gives up to three attempts and CVE-Bench reports one and five.
- Grader. Meta's post says Gemini 1.5 Pro's 61.1% generation rate fell to 5.3% of the 113 Lite samples once patches were checked for semantic equivalence, about 9% of the patches that had built and stopped the crash. Meta writes that "build and crash reproduction are not good enough signals to infer the correctness of generated patches".
- Date and setup. CyberGym's paper reports 17.9% on the full set for its strongest combination without a thinking mode in 2025, and 22.0% for GPT-5 with high reasoning on a 300-instance subset; the leaderboard, whose page says results "are evaluated and submitted by individual teams" and that runs are stochastic, lists entries above 90% in 2026 for different agents and models. Neither is wrong; they describe different systems.
- Version. CVE-Bench replaced arbitrary file upload with remote code execution as one of its eight outcomes in v2.1.0 (12 Jan 2026), after the paper. SWE-bench figures differ by variant: Verified has 500 instances, Lite 300, the full set 2,294.

## Which benchmark answers which question

Limits as stated by the benchmarks' authors, or taken from the tables above. Read 19 Nov 2026.

| Question a security team has | Read | Limit to remember |
|---|---|---|
| Can an agent turn a written report of a memory-safety flaw into a reproducing test? | CyberGym | Reproduction, not repair; leaderboard entries are self-submitted |
| Does an agent's fix keep the service working as well as stopping the exploit? | BountyBench Patch | 25 open-source systems, one gold exploit per bounty |
| Is a native-code patch semantically correct, not just compiling? | AutoPatchBench | C/C++ crashes found by fuzzing |
| Does the model know CWE, CVSS and ATT&CK vocabulary? | CTIBench | Fixed text, 2024 data, English only |
| Can a model reason over malware reports and threat intelligence? | CyberSOCEval | Questions built from reports; not your telemetry |
| Does a coding model suggest insecure code? | CyberSecEval version 1 and 3 | Static analysis patterns give false positives and negatives |
| How quickly could an attacker's agent exploit a new web CVE? | CVE-Bench | Offensive; 40 CVEs, eight attack outcomes |
| What coding-agent lineage is behind a vendor's fix agent? | SWE-bench | No check that a vulnerability is gone |

No benchmark here tests prioritization of a finding against your environment, ticket quality, owner assignment or approval behavior, which are the parts of a security agent's day a team needs to judge. Test those on your own findings with the [evaluation harness](/agent-harness/evaluation-harness/).

## Sources

- [Zhang et al., BountyBench, arXiv 2505.15216 (v3, Dec 2, 2025)](https://arxiv.org/abs/2505.15216), read 19 Nov 2026
- [Alam et al., CTIBench, arXiv 2406.07599 (v3, Nov 11, 2024)](https://arxiv.org/abs/2406.07599), read 19 Nov 2026
- [Zhu et al., CVE-Bench, arXiv 2503.17332 (v4, Jun 24, 2025)](https://arxiv.org/abs/2503.17332), read 19 Nov 2026
- [CVE-Bench repository README (news and version)](https://github.com/uiuc-kang-lab/cve-bench), read 19 Nov 2026
- [Wang et al., CyberGym, arXiv 2506.02548 (v3, Mar 24, 2026)](https://arxiv.org/abs/2506.02548), read 19 Nov 2026
- [CyberGym project page and leaderboard (data file assets/data/cybergym.json)](https://www.cybergym.io/cybergym), read 19 Nov 2026
- [Bhatt et al., Purple Llama CyberSecEval, arXiv 2312.04724](https://arxiv.org/abs/2312.04724), read 19 Nov 2026
- [Wan et al., CyberSecEval 3, arXiv 2408.01605](https://arxiv.org/abs/2408.01605), read 19 Nov 2026
- [CyberSOCEval, arXiv 2509.20166 (Sep 24, 2025)](https://arxiv.org/abs/2509.20166), read 19 Nov 2026
- [Meta Engineering, Introducing AutoPatchBench, 29 Apr 2025](https://engineering.fb.com/2025/04/29/ai-research/autopatchbench-benchmark-ai-powered-security-fixes/), read 19 Nov 2026
- [Jimenez et al., SWE-bench, arXiv 2310.06770](https://arxiv.org/abs/2310.06770), read 19 Nov 2026
- [SWE-bench leaderboards](https://www.swebench.com/), read 19 Nov 2026
- [BountyBench repository README](https://github.com/bountybench/bountybench), read 19 Nov 2026

Last reviewed 19 Nov 2026

Canonical: https://agenticaiincybersecurity.com/benchmarks/
