SWE-bench

Full title: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Authors: Carlos E. Jimenez*, John Yang*, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan (Princeton University / Princeton Language and Intelligence; Pei: University of Chicago) Venue/year: ICLR 2024 (arXiv v3, Nov 2024); leaderboard at swebench.com arXiv: https://arxiv.org/abs/2310.06770 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2310.06770-swe-bench.pdf

Why this paper matters

SWE-bench turned "can LMs code?" into "can LMs do software engineering?" — and in doing so created the yardstick the entire coding-agent field now optimizes. Its 2,294 task instances are not written for the benchmark; they are harvested from real merged pull requests that closed real GitHub issues in 12 popular Python repositories, and graded by the repository's own tests (§1–2). Three properties made it the field's north star: it was brutally hard at release (best model: Claude 2 at 1.96% with BM25 retrieval — abstract), it is execution-graded rather than match-graded, and it is continually updatable — the collection pipeline can ingest new issues created after any model's training cutoff with minimal human effort (§2.3). Every headline agent result since (SWE-agent, Devin, Claude-based coding agents) is a number on this benchmark; understanding exactly what "% resolved" means requires understanding this paper's harness.

The problem

By late 2023, code benchmarks were saturating: HumanEval-style problems are self-contained functions solvable in a few lines, with hand-written specs and tests created for the benchmark (§1, §6). Real software engineering is nothing like that: fixing a bug means navigating a large repository, understanding interactions across functions, classes, and files, and editing multiple locations while breaking nothing (§1). Existing eval suites were both too easy for frontier models and too artificial to predict real-world utility; benchmarks also age badly as their solutions leak into training corpora. The field needed a task source that is challenging, verifiable by execution, realistic, and renewable (§1, §2.3).

The mechanism

Task definition (§2.2)

A task instance = (issue text, codebase snapshot at the PR's base commit). The model must emit a patch file; the harness applies it with unix patch and runs the repository's tests. The metric is % resolved: the fraction of instances where the patch applies and all required tests pass. Average issue: 195.1 words; average codebase: 3,010 non-test files / 438K lines (Table 1) — so the input can never fully fit in context, making retrieval/localization part of the task.

What makes an instance (§2.1, §2.3)

The gold PR averages edits to 1.7 files, 3.0 functions, 32.8 lines (Table 1). Verification signal: every instance has at least one fail-to-pass test — a test that fails before the gold patch and passes after — and 40% have two or more; a median of 51 additional tests check that existing behavior survives (§2.3). Because instances come from independent repos and the pipeline is automatic, the benchmark can be refreshed with post-training-cutoff issues (§2.3, "Continually updatable").

The 12 repositories and their instance counts (Fig. 3) — heavily skewed, which matters when reading per-repo results:

SWE-bench Lite (§2.4) and SWE-Llama (§3)

Lite = 300 instances sampled to be more self-contained, functional-bug-focused, covering 11 of the 12 repos — the affordable split most agent papers report. SWE-Llama 7b/13b are CodeLlama-Python models LoRA-finetuned (attention weights only) on 19,000 issue-PR pairs from 37 disjoint repositories, filtered to ≤30K tokens → 10,000 training sequences; they are the open-model baseline able to consume ~100K-token contexts (§3).

Feeding the model (§4.1)

Since codebases dwarf context windows, the paper defines two retrieval settings: BM25 sparse retrieval (files retrieved against the issue text, under 13K/27K/50K-token budgets) and "oracle" retrieval (the files the gold patch actually edited — explicitly labeled unrealistic, since a real engineer doesn't know them a priori). BM25-27K contains a superset of the oracle files in ~40% of instances, and none of them in almost half (§4.1) — localization failure is baked into the realistic setting.

Context length alone gated who could even play (Table 4):

The algorithm

Two procedures matter: how instances are built, and how a prediction is graded (§2.1; Appendix A.3–A.4).

# ---- TASK CONSTRUCTION (§2.1, Fig. 2; App. A) ----
PRs ← scrape(12 popular PyPI-package repos)          # ~90,000 PRs
for pr in PRs:                                        # Stage II: attributes
    keep iff pr.merged and pr.resolves_issue and pr.edits_tests
for pr in survivors:                                  # Stage III: execution
    C ← checkout(repo, pr.base_commit)                # codebase snapshot
    δ ← pr.patch minus test edits                     # gold solution patch
    T ← pr.test_patch                                 # gold test changes
    log_pre  ← run_tests(C + T)                       # before solution
    log_post ← run_tests(C + δ + T)                   # after solution
    buckets ← compare(log_pre, log_post)              # per-test status →
        {FAIL_TO_PASS, PASS_TO_PASS, FAIL_TO_FAIL, PASS_TO_FAIL}
    keep iff installs & runs cleanly and |FAIL_TO_PASS| ≥ 1
# result: 2,294 instances; buckets cached as ground truth (App. A.3)

# ---- GRADING one model patch δ̂ (App. A.4, Fig. 8) ----
1. reset repo to base commit → C
2. activate the instance's versioned environment
3. install C
4. apply test patch T
5. apply predicted patch δ̂
6.   if apply fails: auto-repair δ̂ (strip context lines, fix headers), retry;
     still failing → score 0
7. run test script → log_δ̂ → test-to-status map
resolved(δ̂) ⇔ every test in FAIL_TO_PASS ∪ PASS_TO_PASS
              appears in log_δ̂ with status "pass"
              # a missing test counts as a fail (App. A.4)
% resolved = mean over the 2,294 instances

Construction, line by line:

• Stage I (scrape) — popular repos are chosen deliberately: popularity proxies for maintenance quality, contributor guidelines, and test coverage (§2.1). ~90,000 PRs go in; 2,294 instances come out (Table 10, App. A).
• Stage II (attribute filter) — cheap metadata checks with one crucial condition: the PR must touch test files. That is what guarantees a behavioral spec exists — the contributed tests encode "what does fixed mean" for this issue (§2.1).
• δ vs T — the PR is split into the solution patch (non-test edits) and the test patch. The model will only ever be graded against T; it never sees δ.
• Stage III (execution filter) — the expensive part: each candidate is installed and run twice — once with only the new tests (some must fail, otherwise the issue was not really unsolved) and once with tests plus solution (they must pass).
• The four buckets — diffing log_pre and log_post classifies every test into FAIL_TO_PASS, PASS_TO_PASS, FAIL_TO_FAIL, PASS_TO_FAIL (App. A.3). The two that matter downstream: FAIL_TO_PASS, the tests certifying the issue is resolved, and PASS_TO_PASS, the regression suite certifying nothing else broke.
• Caching — the buckets are stored as ground truth so grading never needs to re-run the gold patch (App. A.3). Instances whose new tests invoke newly created functions/classes are excluded, so tests cannot trivially mirror one specific implementation (App. A.3).

Grading, line by line:

• Steps 1–4 replay the validated setup (reset, environment, install, test patch) and "reliably do not fail" — any flakiness here was filtered out during construction (App. A.4).
• Steps 5–6 apply the model's patch, with leniency: malformed patches are auto-repaired (context lines stripped, headers recalculated) before giving up. This exists because early LMs were bad at emitting well-formed diffs — which is also why "% apply" is reported separately from "% resolved" in Table 5: formatting, not reasoning, killed many attempts.
• Step 7 + verdict — the strictness lives in the final rule: all FAIL_TO_PASS and all PASS_TO_PASS tests must appear and pass, and a missing test counts as a fail (App. A.4). A model cannot win by deleting tests, skipping them, or fixing the symptom while breaking a neighbor.
• Net meaning — a resolved instance is: real issue, real repo, model patch satisfying the same executable behavioral contract the human fix did.

Results that matter

Main table (BM25 retrieval, Table 5, v3 including Claude 3 Opus):

How resolve rate scales with localization help (Claude 2 unless noted):

Performance scales with how much localization is done for the model — the benchmark's central diagnostic finding. Other results worth retaining: Claude 2's resolve rate drops monotonically as total input length grows, even though longer BM25 budgets retrieve more of the right files (Fig. 5, Table 2 — 1.96 → 1.87 → 1.22 across 13K/27K/50K). Performance on issues from before vs. after 2023 is essentially flat for most models (Table 7), the paper's evidence against training-data contamination driving results. Generating whole files instead of patches is worse (Claude 2: 2.2% vs 4.8% oracle, §5). Fine-tuned models are brittle to context shift: SWE-Llama was trained on oracle-style context and collapses to 0 when BM25 hands it files it should not edit (Table 2 at 50K, §5). Model patches are less than half the length of gold ones (30.1 vs 74.5 total lines, Table 8) — models fix narrowly; humans refactor and anticipate.

Limitations & how to defend the findings

• "1–4% resolve rates — is the benchmark just broken/too hard to measure anything?" The paper's position: difficulty is the point, and the harness reports a smooth spectrum (% apply, oracle, oracle-collapsed) showing models fail progressively at localization → formatting → semantics, not randomly. Fig. 5 and Table 6 turn "too hard" into a diagnosis: context handling is the bottleneck.
• "Aren't the solutions in the training data?" Table 7's before/after-2023 partition shows little performance difference for most models (GPT-4 is the exception, evaluated on a 25% subset), suggesting models don't simply regurgitate seen fixes (§5). The stronger structural defense: the pipeline can always mint new post-cutoff instances (§2.3).
• "Passing tests ≠ good code." Conceded verbatim in §7: execution-based testing "is insufficient to guarantee reliable performance," since model patches can be less comprehensive, efficient, or readable than human ones. The PASS_TO_PASS suite bounds the damage but does not measure quality.
• "The oracle setting is unrealistic." The paper says so itself (§4.1) and reports BM25 as the honest headline; oracle numbers exist to isolate reasoning from retrieval. When quoting SWE-bench numbers, always check which retrieval setting (and which split — full vs Lite) is being cited.
• "Python-only, 12 repos, issue text may over/under-specify." §7 concedes Python-only and hopes to extend; repo skew is visible in Fig. 3 (django alone is 850/2,294 instances). Later community work (e.g., human-validated subsets) refined instance quality — outside this paper, but worth knowing when comparing modern numbers.
• "Tests contributed by the PR might encode the gold patch's exact behavior." Partially mitigated during construction: instances whose new tests reference newly created functions/classes are excluded so tests don't trivially mirror one implementation (App. A.3).

Connections

• Outcome-based evals: SWE-bench is the flagship of grading outcomes (do the tests pass?) over process or text similarity — same philosophy as unit-test-scored code benchmarks, scaled from function to repository.
• SWE-agent: built by the same group explicitly because SWE-bench's own baselines were non-interactive single-shot patch generators (§7 invites "agent-based approaches"); SWE-agent's 12.47% vs Claude 3 Opus RAG's 3.79% quantifies what an agent loop adds on identical tasks.
• Retrieval/RAG: §4.1's BM25-vs-oracle gap is one of the cleanest published demonstrations that localization — not patch writing — is the first-order bottleneck in repo-scale tasks; a lesson driving every agent's search tools since.
• Long-context research: Fig. 5 (performance falls as context grows despite better recall) is an early, task-grounded observation of the "distracted by context" effect later formalized in needle-in-haystack studies.
• Data flywheels: the scrape→filter→execute pipeline doubles as training-data machinery — SWE-bench-train (19K instances) and SWE-Llama (§3) prefigure the issue-PR training corpora behind later coding agents.
• Toolformer/Voyager contrast: those papers propose capabilities; SWE-bench is the measurement instrument — the shelf's example that in agent research the eval harness is itself a first-class artifact, with design decisions (fail-to-pass, pass-to-pass, auto-repair of patches) that shape the whole field's incentives.

Reading guide

1. Figure 1 + §2.1–2.2 — task anatomy and the 3-stage pipeline; 10 minutes, non-skippable.
2. §2.3 features + Table 1 — the statistics that define difficulty (438K-line codebases, 32.8-line gold patches). Table 1 is the one table to study: every later argument about context and localization is latent in it.
3. §4.1 retrieval settings — read carefully; "oracle" vs BM25 is the most-misquoted distinction in the literature.
4. §5 + Tables 5–8, Fig. 5 — results; read the prose paragraphs ("Difficulty correlates with context length", "Generating patches is easier than generating whole files"), they carry the insight.
5. Appendix A.3–A.4 — read if you will ever run or build an eval harness: the FAIL_TO_PASS/PASS_TO_PASS bucketing, the 7-step grading recipe, and the missing-test-counts-as-fail rule live here.
6. Skimmable: §3 (SWE-Llama training details), §6 related work, App. B–F unless debugging specific behaviors.