SWE-bench results and what they measure

SWE-bench results on swebench.com, read 19 Nov 2026, run from 1.96% of issues resolved (Claude 2, Oct 2023) to 79.2% on Verified (two Claude 4.5 Opus entries, Dec 2025). In the shared mini-SWE-agent setup the highest is 76.8% (Claude 4.5 Opus, Feb 2026). Each percent counts issues whose project tests pass, not vulnerabilities removed.

What the benchmark is

SWE-bench gives a model a codebase and an issue description, and the model "is tasked with editing the codebase to address the issue" (Jimenez et al., arXiv 2310.06770, posted 10 Oct 2023). The original set holds 2,294 problems from 12 Python repositories. A patch counts as resolved when the tests that failed before the fix now pass and the tests that passed before still pass. Since 27 Jun 2024 the project runs this in a Docker harness (repository README).

Variants and counts from the swebench.com home page and the Lite, Multilingual and Multimodal pages, read 19 Nov 2026.
VariantInstancesWhat swebench.com says
SWE-bench (Full)2,294The original benchmark: real GitHub issues from 12 Python repositories
Lite300"A subset curated for less costly evaluation"; instances with images, several edited files or more than 3 edit hunks were removed
Verified500"A human-filtered subset" made with OpenAI; released 13 Aug 2024
Multilingual300Tasks from 42 repositories across 9 programming languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust
Multimodal480 (v2, 1 Sep 2026); 517 in the original releaseIssues described with visual elements such as screenshots and mockups

Verified and Multilingual results for the same 13 models

The Verified page runs every language model in one minimal setup, mini-SWE-agent, "No tools, no special scaffold structure; just a simple ReAct agent loop", so these rows compare models rather than scaffolds. The table lists every Verified entry in that setup with release 2.x (the tool-calling version) and the Multilingual entry for the same model. Rows are alphabetical. The site states that release 1.x and 2.x results "are not necessarily comparable", so older bash-only rows are left out.

swebench.com leaderboard data, read 19 Nov 2026. Names are as the site lists them; "(high)" is the site's label on that entry. Verified runs used mini-SWE-agent 2.0.0. Multilingual runs used 2.0.0a0, except GPT 5.2 Codex (2.0.0). The gap is computed here: Verified minus Multilingual.
Verified entryModel ID listedVerified dateVerified % resolvedCost per instance ($)Multilingual dateMultilingual % resolvedGap (points)
Claude 4.5 Haiku (high)claude-haiku-4-5-2025100117 Feb 202666.60.3313 Feb 202664.71.9
Claude 4.5 Opus (high)claude-4-5-opus17 Feb 202676.80.7513 Feb 202670.76.1
Claude 4.5 Sonnet (high)claude-sonnet-4-5-2025092917 Feb 202671.40.6613 Feb 202667.04.4
Claude 4.6 Opusclaude-opus-4-617 Feb 202675.60.5513 Feb 202672.03.6
DeepSeek V3.2 (high)deepseek-v3.217 Feb 202670.00.4513 Feb 202659.011.0
GLM 5 (high)glm-517 Feb 202672.80.5313 Feb 202669.73.1
GPT 5 minigpt-5-mini-2025-08-0717 Feb 202656.20.0513 Feb 202639.716.5
GPT 5.2 (high)gpt-5-217 Feb 202672.80.4713 Feb 202666.76.1
GPT 5.2 Codexgpt-5-2-codex19 Feb 202672.80.4520 Feb 202666.36.5
Gemini 3 Flash (high)gemini-3-flash-preview17 Feb 202675.80.3613 Feb 202672.73.1
Gemini 3 Progemini-3-pro-preview26 Feb 202669.60.9613 Feb 202668.70.9
Kimi K2.5 (high)kimi-k2.517 Feb 202670.80.1513 Feb 202667.33.5
MiniMax M2.5 (high)minimax-m2.517 Feb 202675.80.0716 Feb 202668.37.5

Every model resolves fewer Multilingual tasks than Verified tasks, by 0.9 to 16.5 points. The Multilingual page reports the same pattern: Claude 3.7 Sonnet in SWE-agent resolved 43% there against 63% on Verified, and it attributes the gap to Python-centred tooling. A percent from one variant does not carry to another language mix, which is the situation of a team whose fixes are in Go, Java or JavaScript.

The Full benchmark since 2023

The Full leaderboard lists 24 entries. The 13 below carry the site's flag "Run performed or directly checked by the SWE-bench team", in date order.

swebench.com Full leaderboard, entries flagged as run or checked by the team, read 19 Nov 2026. The 10 Oct 2023 rows are the paper's baselines (retrieval, not an agent).
DateEntry% resolved
10 Oct 2023RAG + Claude 21.96
10 Oct 2023RAG + SWE-Llama 13B0.70
10 Oct 2023RAG + SWE-Llama 7B0.70
10 Oct 2023RAG + ChatGPT 3.50.17
2 Apr 2024SWE-agent + GPT 4 (1106)12.47
2 Apr 2024SWE-agent + Claude 3 Opus10.51
2 Apr 2024RAG + Claude 3 Opus3.79
2 Apr 2024RAG + GPT 4 (1106)1.31
15 Jun 2024AppMap Navie + GPT 4o (13 May 2024)14.60
20 Jun 2024SWE-agent + Claude 3.5 Sonnet18.13
28 Jul 2024SWE-agent + GPT 4o (13 May 2024)11.99
3 Nov 2024OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)29.38
27 Feb 2025SWE-agent 1.0 (Claude 3.7 Sonnet)33.83

The first baseline, Claude 2 at 1.96%, became 33.83% for SWE-agent 1.0 with Claude 3.7 Sonnet on 27 Feb 2025. The change came from both the model and the scaffold around it: in April 2024 the same GPT-4 (1106) resolved 1.31% with retrieval and 12.47% inside SWE-agent. A percent therefore names a model and a scaffold together.

Multimodal: 480 tasks or 517

The home page card and the Multimodal page both give 480 for the current set. The Multimodal page explains: the original release "augmented the benchmark with 517 issues", and "Multimodal v2" of 1 Sep 2026 "retains 480 tasks selected for reproducible evaluation, with known flaky or ungradeable tests removed". The repository README dates the open-source v2 release the same day.

The leaderboard percentages predate v2 (the newest entry is 17 Nov 2025) and fit 517. Multiplying each by 517 gives a whole number of instances within rounding; for example 35.98% is 186 of 517, where 480 would give 172.7. The leaderboard page does not state its denominator, so the 517 reading is inferred here from that arithmetic. Do not compare a v2 run on 480 tasks with these rows.

swebench.com Multimodal leaderboard, all entries dated 28 May 2025 or later, alphabetical, read 19 Nov 2026. Implied instances are computed here as percent times 517.
EntryDateRun or checked by the team% resolvedImplied instances of 517
Codefuse_Pycfuse_SVR17 Nov 2025Yes35.98186
GUIRepair + GPT 4.1 (14 Apr 2025)31 May 2025Yes31.14161
GUIRepair + GPT 4o (6 Aug 2024)31 May 2025Yes30.37157
GUIRepair + o3 (16 Apr 2025)1 Jul 2025Yes35.98186
GUIRepair + o4-mini (16 Apr 2025)31 May 2025Yes33.85175
OpenHands-Versa (Claude-Sonnet 4)28 May 2025Yes34.43178
Refact.ai Agent11 Jun 2025No35.59184

What one percent means in instances and dollars

Take Claude 4.6 Opus (claude-opus-4-6) in mini-SWE-agent 2.0.0 on Verified, 17 Feb 2026: 75.6% of 500 is 378 resolved instances. The site lists a cost of $0.55 per instance, about $276 for the 500. Failed attempts cost the same as successful ones, so each resolved instance cost about $0.73 ($0.5515 divided by 0.756). On the same table MiniMax M2.5 resolved 75.8% at $0.07 per instance and Gemini 3 Pro 69.6% at $0.96. Cost per instance runs from $0.05 (GPT 5 mini) to $0.96 across the 13 rows.

The full Verified table has 180 entries. The largest value, 79.2%, belongs to two entries: Sonar Foundation Agent with Claude 4.5 Opus (5 Dec 2025) and live-SWE-agent with Claude 4.5 Opus, medium effort, build 20251101 (15 Dec 2025). The site does not flag either as run or checked by its team. Treat entries without the flag as submitter claims.

Why a resolved issue is not a fixed vulnerability

The fail-to-pass tests were written for functional bugs. A patch can pass them while the vulnerable path stays reachable from other input. The authors say "relying solely on this method is insufficient to guarantee reliable performance of model generations".

Two security benchmarks grade differently. BountyBench's Patch task restarts the service and applies the original exploit: "If the invariants still pass and the verifier fails, the patch is marked as a success" (Zhang et al.; see BountyBench). Meta's AutoPatchBench adds fuzzing and differential testing to the build-and-crash check, and its case study found most patches that built were not correct (Meta Engineering, 29 Apr 2025).

What SWE-bench does give a security team is a design to copy: a fail-to-pass test plus a regression set, run in a container. Build that from your own past fixes using the evaluation harness, and see the benchmarks index for how the others differ.

Sources

  1. SWE-bench Leaderboards, home page and embedded leaderboard data, read 19 Nov 2026
  2. SWE-bench Verified page, bash-only setup, read 19 Nov 2026
  3. SWE-bench Multilingual page, read 19 Nov 2026
  4. SWE-bench Multimodal page, v2 note of 1 Sep 2026, read 19 Nov 2026
  5. SWE-bench Lite page, read 19 Nov 2026
  6. Jimenez et al., SWE-bench, arXiv 2310.06770, read 19 Nov 2026
  7. SWE-bench repository README, read 19 Nov 2026
  8. Zhang et al., BountyBench, arXiv 2505.15216, read 19 Nov 2026
  9. Meta Engineering, Introducing AutoPatchBench, read 19 Nov 2026

Last reviewed 19 Nov 2026