CyberGym benchmark
CyberGym (arXiv 2506.02548) asks an agent to reproduce 1,507 OSS-Fuzz vulnerabilities in 188 projects from a text description. The strongest paper baseline on the full set (May 2025), OpenHands with Claude Sonnet 4, reached 17.9%; GPT-5 with high reasoning reached 22.0% on a 300-instance subset. The project table (19 Nov 2026) lists 95 entries, 0.1% to 99.4%.
What it measures and how it grades
Six authors led by Zhun Wang, with Dawn Song as last author, posted arXiv 2506.02548 on 3 Jun 2025; v3 followed on 24 Mar 2026. The instances come from OSS-Fuzz, Google's continuous fuzzing service: 1,507 vulnerabilities disclosed between 1 Jan 2017 and 21 Apr 2025, 1,368 of them from the ARVO dataset and 139 newer ones, across 188 projects and 28 sanitizer crash types, mostly C and C++ memory safety bugs.
The agent receives the pre-patch codebase, a text description (median 24 words) and a container where it can submit a test input and see the exit code and output. Success is execution-based: the input must crash the pre-patch program under a sanitizer and must not crash the post-patch program, "which means that the generated PoC accurately reproduces the specific vulnerability that the patch addresses". The metric is the share of instances solved. The agent never gets the patched build.
| Level | What the agent receives | OpenHands with GPT-4.1, May 2025 |
|---|---|---|
| 0 | Pre-patch codebase only, no description (open-ended discovery) | 3.5% |
| 1 | Codebase and text description (the primary task) | 9.4% |
| 2 | Level 1 plus the crash stack trace from the reference input | 13.1% |
| 3 | Level 2 plus the reference patch diff and post-patch codebase | 17.1% |
Published results at Level 1
The paper's baselines ran OpenHands, Codex CLI, Cybench and EnIGMA agents with a 100-iteration cap on checkpoints such as claude-3-7-sonnet-20250219, claude-sonnet-4-20250514, gpt-4.1-2025-04-14, o4-mini-2025-04-16 and gemini-2.5-flash-preview-04-17. The rows below are every Level 1 entry in the project's table dated up to 17 Feb 2026, listed by submitter. Entries are submitted by the teams themselves.
| Submitter | Agent | Model | Date | Trials | Level 1 success |
|---|---|---|---|---|---|
| Anthropic | Anthropic Agent | Claude Opus 4.1 | 29 Sep 2025 | 1 | 25.0% |
| Anthropic | Anthropic Agent | Claude Opus 4.1 | 29 Sep 2025 | 30 | 61.3% |
| Anthropic | Anthropic Agent | Claude Opus 4.5 | 5 Feb 2026 | 1 | 50.6% |
| Anthropic | Anthropic Agent | Claude Opus 4.6 | 5 Feb 2026 | 1 | 66.6% |
| Anthropic | Anthropic Agent | Claude Sonnet 3.7 | 29 Sep 2025 | 1 | 14.5% |
| Anthropic | Anthropic Agent | Claude Sonnet 3.7 | 29 Sep 2025 | 30 | 47.2% |
| Anthropic | Anthropic Agent | Claude Sonnet 4 | 29 Sep 2025 | 1 | 22.6% |
| Anthropic | Anthropic Agent | Claude Sonnet 4 | 29 Sep 2025 | 30 | 59.5% |
| Anthropic | Anthropic Agent | Claude Sonnet 4.5 | 29 Sep 2025 | 1 | 28.9% |
| Anthropic | Anthropic Agent | Claude Sonnet 4.5 | 29 Sep 2025 | 30 | 66.7% |
| Anthropic | Anthropic Agent | Claude Sonnet 4.6 | 17 Feb 2026 | 1 | 65.2% |
| CyberGym Team | Codex CLI | GPT-4.1 | 15 May 2025 | 1 | 7.4% |
| CyberGym Team | Cybench | GPT-4.1 | 15 May 2025 | 1 | 9.0% |
| CyberGym Team | ENiGMA | GPT-4.1 | 15 May 2025 | 1 | 7.2% |
| CyberGym Team | OpenHands | Claude Sonnet 3.7 | 15 May 2025 | 1 | 11.9% |
| CyberGym Team | OpenHands | Claude Sonnet 4 | 23 May 2025 | 1 | 17.9% |
| CyberGym Team | OpenHands | DeepSeek-V3 | 15 May 2025 | 1 | 3.6% |
| CyberGym Team | OpenHands | Gemini 2.5 Flash Preview | 15 May 2025 | 1 | 4.8% |
| CyberGym Team | OpenHands | GPT-4.1 | 15 May 2025 | 1 | 9.4% |
| CyberGym Team | OpenHands | GPT-5 | 5 Dec 2025 | 1 | 39.4% |
| CyberGym Team | OpenHands | o4-mini | 15 May 2025 | 1 | 2.5% |
| CyberGym Team | OpenHands | OpenHands-LM-32B | 15 May 2025 | 1 | 1.7% |
| CyberGym Team | OpenHands | Qwen3-235B-A22B | 15 May 2025 | 1 | 1.9% |
| CyberGym Team | OpenHands | R2E-Gym-32B | 15 May 2025 | 1 | 2.0% |
| CyberGym Team | OpenHands | SWE-Gym-32B | 15 May 2025 | 1 | 0.1% |
| Kimi | Kimi Agent | Kimi K2.5 | 2 Feb 2026 | 1 | 41.3% |
| OpenSage Team | SageAgent | GPT-5 | 9 Feb 2026 | 1 | 60.2% |
| Zhipu AI | Claude Code | GLM-4.7 | 12 Feb 2026 | 1 | 23.5% |
| Zhipu AI | Claude Code | GLM-5 | 12 Feb 2026 | 1 | 43.2% |
Read the Anthropic rows against each other: Claude Sonnet 4.5 reached 28.9% on one trial and 66.7% on 30 trials, the same model on the same date. A rate without its trial count says little.
Other figures from the paper. The union of the eleven models' results solves 27.2% of instances, so their successes overlap little; the union of four agents on GPT-4.1 is 18.4%, nearly double the strongest single agent. Models fine-tuned on SWE-bench-style tasks reached at most 2.0%. The paper tested for contamination by splitting at each model's knowledge cutoff: Claude 3.7 Sonnet solved 11.9% (169 of 1,419) before and 12.5% (11 of 88) after, with p-values above 0.1 for all four models tested.
Where the project table stands now
On 19 Nov 2026 the Level 1 table holds 95 entries dated 15 May 2025 to 24 Oct 2026. Rates run from 0.1% to 99.4%, the median is 84.5%, 37 entries are at or above 90% and 24 are below 50%. All but the four 30-trial Anthropic rows use one trial. The table carries the notice that results are submitted by individual teams, that runs are stochastic, and that, with leading systems already highly successful, small differences between entries may not reflect meaningful capability gaps.
The ten entries with the highest Level 1 rate, plus the April 2026 entries from Anthropic and OpenAI, are below, listed alphabetically by submitter. Every row uses one trial.
| Submitter | Agent | Model | Date | Trials | Level 1 success |
|---|---|---|---|---|---|
| Ansen | Crimson | DeepSeek-V4.1-Flash | 18 Sep 2026 | 1 | 99.0% |
| Anthropic | Anthropic Agent | Claude Mythos Preview | 7 Apr 2026 | 1 | 83.1% |
| Continuum AI | OrcaCyber | Multi-model (DeepSeek-V4-Flash, OrcaCyber-Zero-1.0) | 14 Sep 2026 | 1 | 98.01% |
| Creation Lab | Creation | Multi-model (Creation Model, DeepSeek-V4-Pro, Qwen 3.8 Max) | 7 Sep 2026 | 1 | 98.47% |
| Elliot R&D | elliot-v1 | elliot-rdt-v1 | 6 Oct 2026 | 1 | 97.28% |
| Lyrie | Lyrie Agent | Multi-model (DeepSeek-V4.1-Flash, Abliterated GLM-5.2) | 15 Sep 2026 | 1 | 99.2% |
| OpenAI | OpenAI Agent | GPT-5.5 | 23 Apr 2026 | 1 | 81.8% |
| Ostorlab | Ostorlab Neutron | DeepSeek-V4-Flash | 6 Oct 2026 | 1 | 97.08% |
| Qihoo 360 | TuLongFeng | DeepSeek-V4.1-Flash | 23 Sep 2026 | 1 | 99.38% |
| RO0T Lab | RO0T Agent | DeepSeek-V4.1-Flash | 18 Sep 2026 | 1 | 97.7% |
| Sangfor AI | Sangfor AI | GLM-5.3 | 2 Sep 2026 | 1 | 97.21% |
| VARAS | VARAS-OneMind | DeepSeek-V4-Flash | 16 Sep 2026 | 1 | 96.95% |
The submission guidelines dated 4 Aug 2026 say the most capable models under unconstrained resources have nearly saturated the benchmark, so future submissions must report cost alongside success rate; the report schema asks for tokens, time and model requests per task. The FAQ requires one designated final submission per instance, because an any-of count "increasingly rewards brute-forcing", and asks submitters to state network access and whether the agent had a runnable vulnerable image.
The results data file has no cost field. Four 2026 write-ups linked from the table report cost per task for their Level 1 runs, each by its own method, listed here alphabetically by submitter.
| Submitter and agent | Model | Date | Level 1 success | Reported cost per task |
|---|---|---|---|---|
| Aeye Security Lab, Claude Code | ASL-Cyber-Flash (finetuned from DeepSeek-V4-Flash) | 3 Sep 2026 | 85.87% | about $0.16 (about $240 of GPU time for 1,507 tasks) |
| Fangcun AI, FangcunCyber | DeepSeek-V4-Flash | 16 Aug 2026 | 91.31% | $0.91 estimated, 56.3 minutes mean |
| Ostorlab, Neutron | DeepSeek-V4-Flash | 6 Oct 2026 | 97.08% | $1.04 average inference cost |
| Sealed Realm Team, Ji Realm | DeepSeek-V4-Flash and GLM-5.2 | 7 Sep 2026 | 92.44% | $0.896 estimated infrastructure cost |
Cost to run it: the paper reports over $40,000 of API credits and 1,000 H100 GPU hours for its full study, about $3,000 for one non-thinking pass over all 1,507 instances, and a random 300-instance subset (about 20%) for cheaper runs. The paper's agent comparison capped spending near $2.0 per task. The same runs found 34 previously unknown vulnerabilities and 18 incomplete patches in 15 projects, of which 4 received CVEs and 10 vulnerabilities were patched when written.
What it does not measure
- Anything beyond sanitizer-detectable memory safety bugs in mostly C and C++ code. The authors state this scope; web, logic and configuration flaws are outside it.
- Prioritization, exposure in your estate, or whether a fix is safe. Success means a crashing input exists, not that the bug matters to you.
- Writing the fix. Level 3 hands the agent the patch; the reproduction task never asks for one.
- Approval behavior or tool use under your permissions.
How a security team would use it
Level 1 is the closest public test of a step teams do daily: deciding whether a written report describes a real, reachable defect. Read a Level 1 rate as the chance that an agent with that setup reproduces a typical open-source memory bug from a short description in a lab. It is not the chance of reproducing a report about your own product, whose build and inputs the agent may not have.
A priced example from the paper's own numbers: at the $2.0 per task budget and the 9.4% rate of OpenHands with GPT-4.1, and assuming the $2.0 average applies to that run and each failed attempt spends it, one reproduction costs about $2.0 / 0.094 = $21. At 17.9% it would cost about $11. Compare that with the roughly 5 hours the paper cites for a human expert to reproduce a known bug. A failed attempt proves little, since the bugs in the set are all real; a successful one is evidence for the validation step in vulnerability triage and remediation.
With the highest reported entries near 99%, a Level 1 rate no longer separates strong agents. Test on a set drawn from your own reports instead, using the evaluation harness, and keep any reproduction server on an isolated network as the project's README requires. Neighbors: CVE-Bench for web exploitation, BountyBench for patching, the benchmarks index.
Sources
- Wang et al., CyberGym, arXiv 2506.02548 (v3, 24 Mar 2026), read 19 Nov 2026
- CyberGym full text: Sections 3 to 5, Tables 2 and 6, read 19 Nov 2026
- CyberGym project page and Level 1 results table, read 19 Nov 2026
- CyberGym results data, Levels 0 to 3, read 19 Nov 2026
- CyberGym repository README, FAQ.md and SUBMISSION.md (4 Aug 2026), read 19 Nov 2026
- Frontier AI Cybersecurity Observatory home page, read 19 Nov 2026
- CyberGym SUBMISSION.md, report schema with cost fields (4 Aug 2026), read 19 Nov 2026
- FangcunCyber CyberGym write-up, read 19 Nov 2026
- ASL-Cyber-Flash submission report, read 19 Nov 2026
- Ji Realm CyberGym write-up, read 19 Nov 2026
- Ostorlab Neutron CyberGym technical write-up, read 19 Nov 2026
Last reviewed 19 Nov 2026