CyberGym benchmark

CyberGym (arXiv 2506.02548) asks an agent to reproduce 1,507 OSS-Fuzz vulnerabilities in 188 projects from a text description. The strongest paper baseline on the full set (May 2025), OpenHands with Claude Sonnet 4, reached 17.9%; GPT-5 with high reasoning reached 22.0% on a 300-instance subset. The project table (19 Nov 2026) lists 95 entries, 0.1% to 99.4%.

What it measures and how it grades

Six authors led by Zhun Wang, with Dawn Song as last author, posted arXiv 2506.02548 on 3 Jun 2025; v3 followed on 24 Mar 2026. The instances come from OSS-Fuzz, Google's continuous fuzzing service: 1,507 vulnerabilities disclosed between 1 Jan 2017 and 21 Apr 2025, 1,368 of them from the ARVO dataset and 139 newer ones, across 188 projects and 28 sanitizer crash types, mostly C and C++ memory safety bugs.

The agent receives the pre-patch codebase, a text description (median 24 words) and a container where it can submit a test input and see the exit code and output. Success is execution-based: the input must crash the pre-patch program under a sanitizer and must not crash the post-patch program, "which means that the generated PoC accurately reproduces the specific vulnerability that the patch addresses". The metric is the share of instances solved. The agent never gets the patched build.

Levels from Section 3.2 of the paper; rates from the project's results data, read 19 Nov 2026.
LevelWhat the agent receivesOpenHands with GPT-4.1, May 2025
0Pre-patch codebase only, no description (open-ended discovery)3.5%
1Codebase and text description (the primary task)9.4%
2Level 1 plus the crash stack trace from the reference input13.1%
3Level 2 plus the reference patch diff and post-patch codebase17.1%

Published results at Level 1

The paper's baselines ran OpenHands, Codex CLI, Cybench and EnIGMA agents with a 100-iteration cap on checkpoints such as claude-3-7-sonnet-20250219, claude-sonnet-4-20250514, gpt-4.1-2025-04-14, o4-mini-2025-04-16 and gemini-2.5-flash-preview-04-17. The rows below are every Level 1 entry in the project's table dated up to 17 Feb 2026, listed by submitter. Entries are submitted by the teams themselves.

Source: cybergym.io project table (assets/data/cybergym.json), read 19 Nov 2026. Trials are attempts per instance; an instance counts as solved if any trial succeeds. The 5 Dec 2025 GPT-5 row is a later run than the paper's GPT-5 figures of 7.7% at minimal reasoning and 22.0% at high reasoning on the 300-instance subset.
SubmitterAgentModelDateTrialsLevel 1 success
AnthropicAnthropic AgentClaude Opus 4.129 Sep 2025125.0%
AnthropicAnthropic AgentClaude Opus 4.129 Sep 20253061.3%
AnthropicAnthropic AgentClaude Opus 4.55 Feb 2026150.6%
AnthropicAnthropic AgentClaude Opus 4.65 Feb 2026166.6%
AnthropicAnthropic AgentClaude Sonnet 3.729 Sep 2025114.5%
AnthropicAnthropic AgentClaude Sonnet 3.729 Sep 20253047.2%
AnthropicAnthropic AgentClaude Sonnet 429 Sep 2025122.6%
AnthropicAnthropic AgentClaude Sonnet 429 Sep 20253059.5%
AnthropicAnthropic AgentClaude Sonnet 4.529 Sep 2025128.9%
AnthropicAnthropic AgentClaude Sonnet 4.529 Sep 20253066.7%
AnthropicAnthropic AgentClaude Sonnet 4.617 Feb 2026165.2%
CyberGym TeamCodex CLIGPT-4.115 May 202517.4%
CyberGym TeamCybenchGPT-4.115 May 202519.0%
CyberGym TeamENiGMAGPT-4.115 May 202517.2%
CyberGym TeamOpenHandsClaude Sonnet 3.715 May 2025111.9%
CyberGym TeamOpenHandsClaude Sonnet 423 May 2025117.9%
CyberGym TeamOpenHandsDeepSeek-V315 May 202513.6%
CyberGym TeamOpenHandsGemini 2.5 Flash Preview15 May 202514.8%
CyberGym TeamOpenHandsGPT-4.115 May 202519.4%
CyberGym TeamOpenHandsGPT-55 Dec 2025139.4%
CyberGym TeamOpenHandso4-mini15 May 202512.5%
CyberGym TeamOpenHandsOpenHands-LM-32B15 May 202511.7%
CyberGym TeamOpenHandsQwen3-235B-A22B15 May 202511.9%
CyberGym TeamOpenHandsR2E-Gym-32B15 May 202512.0%
CyberGym TeamOpenHandsSWE-Gym-32B15 May 202510.1%
KimiKimi AgentKimi K2.52 Feb 2026141.3%
OpenSage TeamSageAgentGPT-59 Feb 2026160.2%
Zhipu AIClaude CodeGLM-4.712 Feb 2026123.5%
Zhipu AIClaude CodeGLM-512 Feb 2026143.2%

Read the Anthropic rows against each other: Claude Sonnet 4.5 reached 28.9% on one trial and 66.7% on 30 trials, the same model on the same date. A rate without its trial count says little.

Other figures from the paper. The union of the eleven models' results solves 27.2% of instances, so their successes overlap little; the union of four agents on GPT-4.1 is 18.4%, nearly double the strongest single agent. Models fine-tuned on SWE-bench-style tasks reached at most 2.0%. The paper tested for contamination by splitting at each model's knowledge cutoff: Claude 3.7 Sonnet solved 11.9% (169 of 1,419) before and 12.5% (11 of 88) after, with p-values above 0.1 for all four models tested.

Where the project table stands now

On 19 Nov 2026 the Level 1 table holds 95 entries dated 15 May 2025 to 24 Oct 2026. Rates run from 0.1% to 99.4%, the median is 84.5%, 37 entries are at or above 90% and 24 are below 50%. All but the four 30-trial Anthropic rows use one trial. The table carries the notice that results are submitted by individual teams, that runs are stochastic, and that, with leading systems already highly successful, small differences between entries may not reflect meaningful capability gaps.

The ten entries with the highest Level 1 rate, plus the April 2026 entries from Anthropic and OpenAI, are below, listed alphabetically by submitter. Every row uses one trial.

Source: cybergym.io project table (assets/data/cybergym.json), read 19 Nov 2026. Entries are submitted by the teams themselves.
SubmitterAgentModelDateTrialsLevel 1 success
AnsenCrimsonDeepSeek-V4.1-Flash18 Sep 2026199.0%
AnthropicAnthropic AgentClaude Mythos Preview7 Apr 2026183.1%
Continuum AIOrcaCyberMulti-model (DeepSeek-V4-Flash, OrcaCyber-Zero-1.0)14 Sep 2026198.01%
Creation LabCreationMulti-model (Creation Model, DeepSeek-V4-Pro, Qwen 3.8 Max)7 Sep 2026198.47%
Elliot R&Delliot-v1elliot-rdt-v16 Oct 2026197.28%
LyrieLyrie AgentMulti-model (DeepSeek-V4.1-Flash, Abliterated GLM-5.2)15 Sep 2026199.2%
OpenAIOpenAI AgentGPT-5.523 Apr 2026181.8%
OstorlabOstorlab NeutronDeepSeek-V4-Flash6 Oct 2026197.08%
Qihoo 360TuLongFengDeepSeek-V4.1-Flash23 Sep 2026199.38%
RO0T LabRO0T AgentDeepSeek-V4.1-Flash18 Sep 2026197.7%
Sangfor AISangfor AIGLM-5.32 Sep 2026197.21%
VARASVARAS-OneMindDeepSeek-V4-Flash16 Sep 2026196.95%

The submission guidelines dated 4 Aug 2026 say the most capable models under unconstrained resources have nearly saturated the benchmark, so future submissions must report cost alongside success rate; the report schema asks for tokens, time and model requests per task. The FAQ requires one designated final submission per instance, because an any-of count "increasingly rewards brute-forcing", and asks submitters to state network access and whether the agent had a runnable vulnerable image.

The results data file has no cost field. Four 2026 write-ups linked from the table report cost per task for their Level 1 runs, each by its own method, listed here alphabetically by submitter.

Level 1 success from cybergym.io data; costs from each submitter's write-up (the schema's est_usd_cost is optional and each estimate uses its own pricing), read 19 Nov 2026.
Submitter and agentModelDateLevel 1 successReported cost per task
Aeye Security Lab, Claude CodeASL-Cyber-Flash (finetuned from DeepSeek-V4-Flash)3 Sep 202685.87%about $0.16 (about $240 of GPU time for 1,507 tasks)
Fangcun AI, FangcunCyberDeepSeek-V4-Flash16 Aug 202691.31%$0.91 estimated, 56.3 minutes mean
Ostorlab, NeutronDeepSeek-V4-Flash6 Oct 202697.08%$1.04 average inference cost
Sealed Realm Team, Ji RealmDeepSeek-V4-Flash and GLM-5.27 Sep 202692.44%$0.896 estimated infrastructure cost

Cost to run it: the paper reports over $40,000 of API credits and 1,000 H100 GPU hours for its full study, about $3,000 for one non-thinking pass over all 1,507 instances, and a random 300-instance subset (about 20%) for cheaper runs. The paper's agent comparison capped spending near $2.0 per task. The same runs found 34 previously unknown vulnerabilities and 18 incomplete patches in 15 projects, of which 4 received CVEs and 10 vulnerabilities were patched when written.

What it does not measure

  • Anything beyond sanitizer-detectable memory safety bugs in mostly C and C++ code. The authors state this scope; web, logic and configuration flaws are outside it.
  • Prioritization, exposure in your estate, or whether a fix is safe. Success means a crashing input exists, not that the bug matters to you.
  • Writing the fix. Level 3 hands the agent the patch; the reproduction task never asks for one.
  • Approval behavior or tool use under your permissions.

How a security team would use it

Level 1 is the closest public test of a step teams do daily: deciding whether a written report describes a real, reachable defect. Read a Level 1 rate as the chance that an agent with that setup reproduces a typical open-source memory bug from a short description in a lab. It is not the chance of reproducing a report about your own product, whose build and inputs the agent may not have.

A priced example from the paper's own numbers: at the $2.0 per task budget and the 9.4% rate of OpenHands with GPT-4.1, and assuming the $2.0 average applies to that run and each failed attempt spends it, one reproduction costs about $2.0 / 0.094 = $21. At 17.9% it would cost about $11. Compare that with the roughly 5 hours the paper cites for a human expert to reproduce a known bug. A failed attempt proves little, since the bugs in the set are all real; a successful one is evidence for the validation step in vulnerability triage and remediation.

With the highest reported entries near 99%, a Level 1 rate no longer separates strong agents. Test on a set drawn from your own reports instead, using the evaluation harness, and keep any reproduction server on an isolated network as the project's README requires. Neighbors: CVE-Bench for web exploitation, BountyBench for patching, the benchmarks index.

Sources

  1. Wang et al., CyberGym, arXiv 2506.02548 (v3, 24 Mar 2026), read 19 Nov 2026
  2. CyberGym full text: Sections 3 to 5, Tables 2 and 6, read 19 Nov 2026
  3. CyberGym project page and Level 1 results table, read 19 Nov 2026
  4. CyberGym results data, Levels 0 to 3, read 19 Nov 2026
  5. CyberGym repository README, FAQ.md and SUBMISSION.md (4 Aug 2026), read 19 Nov 2026
  6. Frontier AI Cybersecurity Observatory home page, read 19 Nov 2026
  7. CyberGym SUBMISSION.md, report schema with cost fields (4 Aug 2026), read 19 Nov 2026
  8. FangcunCyber CyberGym write-up, read 19 Nov 2026
  9. ASL-Cyber-Flash submission report, read 19 Nov 2026
  10. Ji Realm CyberGym write-up, read 19 Nov 2026
  11. Ostorlab Neutron CyberGym technical write-up, read 19 Nov 2026

Last reviewed 19 Nov 2026