SWE-bench results and what they measure
SWE-bench results on swebench.com, read 19 Nov 2026, run from 1.96% of issues resolved (Claude 2, Oct 2023) to 79.2% on Verified (two Claude 4.5 Opus entries, Dec 2025). In the shared mini-SWE-agent setup the highest is 76.8% (Claude 4.5 Opus, Feb 2026). Each percent counts issues whose project tests pass, not vulnerabilities removed.
What the benchmark is
SWE-bench gives a model a codebase and an issue description, and the model "is tasked with editing the codebase to address the issue" (Jimenez et al., arXiv 2310.06770, posted 10 Oct 2023). The original set holds 2,294 problems from 12 Python repositories. A patch counts as resolved when the tests that failed before the fix now pass and the tests that passed before still pass. Since 27 Jun 2024 the project runs this in a Docker harness (repository README).
| Variant | Instances | What swebench.com says |
|---|---|---|
| SWE-bench (Full) | 2,294 | The original benchmark: real GitHub issues from 12 Python repositories |
| Lite | 300 | "A subset curated for less costly evaluation"; instances with images, several edited files or more than 3 edit hunks were removed |
| Verified | 500 | "A human-filtered subset" made with OpenAI; released 13 Aug 2024 |
| Multilingual | 300 | Tasks from 42 repositories across 9 programming languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust |
| Multimodal | 480 (v2, 1 Sep 2026); 517 in the original release | Issues described with visual elements such as screenshots and mockups |
Verified and Multilingual results for the same 13 models
The Verified page runs every language model in one minimal setup, mini-SWE-agent, "No tools, no special scaffold structure; just a simple ReAct agent loop", so these rows compare models rather than scaffolds. The table lists every Verified entry in that setup with release 2.x (the tool-calling version) and the Multilingual entry for the same model. Rows are alphabetical. The site states that release 1.x and 2.x results "are not necessarily comparable", so older bash-only rows are left out.
| Verified entry | Model ID listed | Verified date | Verified % resolved | Cost per instance ($) | Multilingual date | Multilingual % resolved | Gap (points) |
|---|---|---|---|---|---|---|---|
| Claude 4.5 Haiku (high) | claude-haiku-4-5-20251001 | 17 Feb 2026 | 66.6 | 0.33 | 13 Feb 2026 | 64.7 | 1.9 |
| Claude 4.5 Opus (high) | claude-4-5-opus | 17 Feb 2026 | 76.8 | 0.75 | 13 Feb 2026 | 70.7 | 6.1 |
| Claude 4.5 Sonnet (high) | claude-sonnet-4-5-20250929 | 17 Feb 2026 | 71.4 | 0.66 | 13 Feb 2026 | 67.0 | 4.4 |
| Claude 4.6 Opus | claude-opus-4-6 | 17 Feb 2026 | 75.6 | 0.55 | 13 Feb 2026 | 72.0 | 3.6 |
| DeepSeek V3.2 (high) | deepseek-v3.2 | 17 Feb 2026 | 70.0 | 0.45 | 13 Feb 2026 | 59.0 | 11.0 |
| GLM 5 (high) | glm-5 | 17 Feb 2026 | 72.8 | 0.53 | 13 Feb 2026 | 69.7 | 3.1 |
| GPT 5 mini | gpt-5-mini-2025-08-07 | 17 Feb 2026 | 56.2 | 0.05 | 13 Feb 2026 | 39.7 | 16.5 |
| GPT 5.2 (high) | gpt-5-2 | 17 Feb 2026 | 72.8 | 0.47 | 13 Feb 2026 | 66.7 | 6.1 |
| GPT 5.2 Codex | gpt-5-2-codex | 19 Feb 2026 | 72.8 | 0.45 | 20 Feb 2026 | 66.3 | 6.5 |
| Gemini 3 Flash (high) | gemini-3-flash-preview | 17 Feb 2026 | 75.8 | 0.36 | 13 Feb 2026 | 72.7 | 3.1 |
| Gemini 3 Pro | gemini-3-pro-preview | 26 Feb 2026 | 69.6 | 0.96 | 13 Feb 2026 | 68.7 | 0.9 |
| Kimi K2.5 (high) | kimi-k2.5 | 17 Feb 2026 | 70.8 | 0.15 | 13 Feb 2026 | 67.3 | 3.5 |
| MiniMax M2.5 (high) | minimax-m2.5 | 17 Feb 2026 | 75.8 | 0.07 | 16 Feb 2026 | 68.3 | 7.5 |
Every model resolves fewer Multilingual tasks than Verified tasks, by 0.9 to 16.5 points. The Multilingual page reports the same pattern: Claude 3.7 Sonnet in SWE-agent resolved 43% there against 63% on Verified, and it attributes the gap to Python-centred tooling. A percent from one variant does not carry to another language mix, which is the situation of a team whose fixes are in Go, Java or JavaScript.
The Full benchmark since 2023
The Full leaderboard lists 24 entries. The 13 below carry the site's flag "Run performed or directly checked by the SWE-bench team", in date order.
| Date | Entry | % resolved |
|---|---|---|
| 10 Oct 2023 | RAG + Claude 2 | 1.96 |
| 10 Oct 2023 | RAG + SWE-Llama 13B | 0.70 |
| 10 Oct 2023 | RAG + SWE-Llama 7B | 0.70 |
| 10 Oct 2023 | RAG + ChatGPT 3.5 | 0.17 |
| 2 Apr 2024 | SWE-agent + GPT 4 (1106) | 12.47 |
| 2 Apr 2024 | SWE-agent + Claude 3 Opus | 10.51 |
| 2 Apr 2024 | RAG + Claude 3 Opus | 3.79 |
| 2 Apr 2024 | RAG + GPT 4 (1106) | 1.31 |
| 15 Jun 2024 | AppMap Navie + GPT 4o (13 May 2024) | 14.60 |
| 20 Jun 2024 | SWE-agent + Claude 3.5 Sonnet | 18.13 |
| 28 Jul 2024 | SWE-agent + GPT 4o (13 May 2024) | 11.99 |
| 3 Nov 2024 | OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | 29.38 |
| 27 Feb 2025 | SWE-agent 1.0 (Claude 3.7 Sonnet) | 33.83 |
The first baseline, Claude 2 at 1.96%, became 33.83% for SWE-agent 1.0 with Claude 3.7 Sonnet on 27 Feb 2025. The change came from both the model and the scaffold around it: in April 2024 the same GPT-4 (1106) resolved 1.31% with retrieval and 12.47% inside SWE-agent. A percent therefore names a model and a scaffold together.
Multimodal: 480 tasks or 517
The home page card and the Multimodal page both give 480 for the current set. The Multimodal page explains: the original release "augmented the benchmark with 517 issues", and "Multimodal v2" of 1 Sep 2026 "retains 480 tasks selected for reproducible evaluation, with known flaky or ungradeable tests removed". The repository README dates the open-source v2 release the same day.
The leaderboard percentages predate v2 (the newest entry is 17 Nov 2025) and fit 517. Multiplying each by 517 gives a whole number of instances within rounding; for example 35.98% is 186 of 517, where 480 would give 172.7. The leaderboard page does not state its denominator, so the 517 reading is inferred here from that arithmetic. Do not compare a v2 run on 480 tasks with these rows.
| Entry | Date | Run or checked by the team | % resolved | Implied instances of 517 |
|---|---|---|---|---|
| Codefuse_Pycfuse_SVR | 17 Nov 2025 | Yes | 35.98 | 186 |
| GUIRepair + GPT 4.1 (14 Apr 2025) | 31 May 2025 | Yes | 31.14 | 161 |
| GUIRepair + GPT 4o (6 Aug 2024) | 31 May 2025 | Yes | 30.37 | 157 |
| GUIRepair + o3 (16 Apr 2025) | 1 Jul 2025 | Yes | 35.98 | 186 |
| GUIRepair + o4-mini (16 Apr 2025) | 31 May 2025 | Yes | 33.85 | 175 |
| OpenHands-Versa (Claude-Sonnet 4) | 28 May 2025 | Yes | 34.43 | 178 |
| Refact.ai Agent | 11 Jun 2025 | No | 35.59 | 184 |
What one percent means in instances and dollars
Take Claude 4.6 Opus (claude-opus-4-6) in mini-SWE-agent 2.0.0 on Verified, 17 Feb 2026: 75.6% of 500 is 378 resolved instances. The site lists a cost of $0.55 per instance, about $276 for the 500. Failed attempts cost the same as successful ones, so each resolved instance cost about $0.73 ($0.5515 divided by 0.756). On the same table MiniMax M2.5 resolved 75.8% at $0.07 per instance and Gemini 3 Pro 69.6% at $0.96. Cost per instance runs from $0.05 (GPT 5 mini) to $0.96 across the 13 rows.
The full Verified table has 180 entries. The largest value, 79.2%, belongs to two entries: Sonar Foundation Agent with Claude 4.5 Opus (5 Dec 2025) and live-SWE-agent with Claude 4.5 Opus, medium effort, build 20251101 (15 Dec 2025). The site does not flag either as run or checked by its team. Treat entries without the flag as submitter claims.
Why a resolved issue is not a fixed vulnerability
The fail-to-pass tests were written for functional bugs. A patch can pass them while the vulnerable path stays reachable from other input. The authors say "relying solely on this method is insufficient to guarantee reliable performance of model generations".
Two security benchmarks grade differently. BountyBench's Patch task restarts the service and applies the original exploit: "If the invariants still pass and the verifier fails, the patch is marked as a success" (Zhang et al.; see BountyBench). Meta's AutoPatchBench adds fuzzing and differential testing to the build-and-crash check, and its case study found most patches that built were not correct (Meta Engineering, 29 Apr 2025).
What SWE-bench does give a security team is a design to copy: a fail-to-pass test plus a regression set, run in a container. Build that from your own past fixes using the evaluation harness, and see the benchmarks index for how the others differ.
Sources
- SWE-bench Leaderboards, home page and embedded leaderboard data, read 19 Nov 2026
- SWE-bench Verified page, bash-only setup, read 19 Nov 2026
- SWE-bench Multilingual page, read 19 Nov 2026
- SWE-bench Multimodal page, v2 note of 1 Sep 2026, read 19 Nov 2026
- SWE-bench Lite page, read 19 Nov 2026
- Jimenez et al., SWE-bench, arXiv 2310.06770, read 19 Nov 2026
- SWE-bench repository README, read 19 Nov 2026
- Zhang et al., BountyBench, arXiv 2505.15216, read 19 Nov 2026
- Meta Engineering, Introducing AutoPatchBench, read 19 Nov 2026
Last reviewed 19 Nov 2026