SWE-agent

Full title: SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering Authors: John Yang*, Carlos E. Jimenez*, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press (Princeton Language and Intelligence, Princeton University) Venue/year: NeurIPS 2024 (arXiv May 2024, v3 Nov 2024); code/leaderboard at swe-agent.com arXiv: https://arxiv.org/abs/2405.15793 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2405.15793-swe-agent.pdf

Why this paper matters

SWE-agent's contribution is a concept, not just a system: the agent-computer interface (ACI) — the thesis that "LM agents represent a new category of end users, with their own needs and abilities" (§2), and that the interface between agent and computer should be designed for them the way IDEs are designed for humans. Holding the LM fixed and redesigning only its commands and feedback took SWE-bench performance from 3.8% (best prior non-interactive system) to 12.47% with GPT-4 Turbo (§1) — a 3× jump attributable purely to interface design. This paper is why every modern coding harness (Claude Code included) looks the way it does: windowed file viewers, guarded string editors, capped search results, collapsed observation history. It reframed agent capability as co-produced by model and environment, and made the environment an object of empirical study with its own ablations.

The problem

Agents dropped into a raw Linux shell struggle in characteristic ways: the shell offers no feedback when an edit is invalid, its commands (dozens of flags each) were designed for humans who can flexibly ignore irrelevant output, and multi-step operations like "view part of a file, change three lines" require fragile compositions of cd/ls/cat/sed (§1, §2, §5.1). LMs, unlike humans, pay a fixed context/compute cost for every token of output and cannot visually skim; verbose or uninformative feedback is actively harmful (§2). Meanwhile the existing SWE-bench baselines were non-interactive RAG pipelines — retrieve files, emit one patch, no execution, no iteration — capping performance at a few percent (§1, §4). Missing was a systematic answer to: what interface makes an LM effective at operating a computer?

The mechanism

ACI design principles (§2)

Derived from manually inspecting agent behavior on a dev set plus a grid search over configurations, four principles:

1. Actions should be simple and easy to understand — few options, concise documentation; no flag jungles.
2. Actions should be compact and efficient — important operations (navigate, edit) consolidated into single actions; no multi-turn compositions for one logical step.
3. Environment feedback should be informative and concise — show the effect of an action (e.g., updated file content after an edit) without flooding context.
4. Guardrails mitigate error propagation — e.g., a syntax linter that blocks bad edits before they land, because LMs recover poorly from self-inflicted errors.

The interface (§3)

Built on top of the Linux shell (ordinary bash remains available), SWE-agent adds:

• Search/navigation: find_file (filenames), search_file / search_dir (strings) return summarized results, capped at 50 hits — beyond that the agent is told to narrow the query rather than being flooded.
• File viewer: open shows a window of at most 100 lines with line numbers, plus full path, total line count, and how many lines lie above/below; scroll_up/scroll_down/goto move the window (Fig. 3a).
• File editor: edit start:end + replacement text rewrites a line range in the open file; the viewer immediately redisplays the result. A code linter vets every edit: on error, the edit is discarded and the agent is shown the error with before/after snippets, then asked to retry (Fig. 3b, §3).
• Context management: the agent emits a ReAct-style thought + action each turn; malformed outputs trigger a retry loop; empty output produces an explicit "Your command ran successfully and did not produce any output" message; all observations except the last 5 are collapsed to a single line; of repeated error messages only the first is kept (§3).

Everything was tuned on SWE-bench's development split via qualitative trajectory analysis plus a sweep over window size, history processing, and temperature (§4, App. B.1).

Experimental frame (§4)

• Models: GPT-4 Turbo (gpt-4-1106-preview, 128K context) and Claude 3 Opus (claude-3-opus-20240229, 200K). Open models (Llama 3, DeepSeek Coder) were tried and dropped — subpar agent behavior, and 8K-class context windows leave no room for multi-turn interaction (§4).
• Baselines: the non-interactive RAG setting from the SWE-bench paper, and "Shell-only" — the same interactive loop but with only the raw Linux shell (adapted from InterCode), isolating the ACI's contribution (§4).
• Metrics: % resolved (all fail-to-pass and pass-to-pass tests green), plus average API dollar cost per instance; per-instance budget $4, with edits auto-submitted at budget exhaustion (§4).
• Main-result protocol: full SWE-bench test set for headline numbers; ablations and analyses on the 300-instance Lite split (§4).

The algorithm

Reconstructed from §3 and §4:

# ---- THE ACI COMMAND SET (§3) ----
search : find_file <name> [dir] | search_file <str> [file] | search_dir <str> [dir]
         → summarized matches; > 50 hits → "please narrow your query"
view   : open <path> [line] | goto <line> | scroll_up | scroll_down
         → ≤100-line window + path, total lines, #lines above/below
edit   : edit <start>:<end>
             <replacement lines>
         end_of_edit
         → linted first: syntax error ⇒ edit DISCARDED, error + context
           shown, retry; success ⇒ viewer redisplays updated window
shell  : any bash command (python, pytest, ls, ...)
submit : end episode, emit current repo diff as the prediction

# ---- THE AGENT LOOP (§3 "context management", §4) ----
history ← [system prompt + ACI docs, (optional) demo trajectory, issue]
repeat:
    (thought, action) ← LM(history)      # ReAct; malformed → error, retry
    obs ← ACI.execute(action)            # incl. "no output" placeholder
    history ← compress(history):         # keep the plan, drop the bulk
        - observations older than the last 5 → 1 line each
        - duplicate error messages → keep only the first
    append(thought, action, obs)
until action == submit or cost > $4      # budget ⇒ auto-submit edits
prediction ← final repository diff; graded by SWE-bench harness

Line by line — the command set is the contribution; each line encodes one design principle:

• search_* with a 50-hit cap — exists because raw grep/find "occasionally produce many lines of irrelevant results" (§5.1). Overflow is converted into a corrective instruction ("narrow your query") instead of a context flood — principle 3 applied to search.
• Summarized, not paginated, results — showing all matches at once beats a Vim-style next/prev iterator, because agents compulsively page through every result before acting (§5.1, Fig. 5).
• open with a 100-line window — solves the "LMs can't skim" problem. 100 lines is the empirical sweet spot: 30 lines starves localization, whole-file display drowns it (Table 3).
• edit start:end + end_of_edit — consolidates a whole IDE gesture (locate, replace range, re-view) into one action with immediate feedback (principle 2); the viewer auto-redisplays so the agent sees the effect of its edit (principle 3).
• The linter guardrail — an edit that introduces a syntax error is rejected before it lands, with before/after context shown (principle 4). This matters because self-inflicted errors cascade: an editing attempt has a 90.5% chance of eventual success, but only 57.2% after a single failed edit (§5.2).
• thought + action each turn — plain ReAct (§3); malformed generations get an error and a retry, so the transcript stays parseable.
• "No output" placeholder — silence is ambiguous to an LM; the explicit success message removes a whole class of misinterpretation (§3).
• Observation collapsing — all but the last 5 observations shrink to one line each: the thought/action skeleton (the plan) is retained while stale file views are discarded, buying more turns within the context window (§3).
• Budget stop — the loop ends at submit or the $4 per-instance cap, at which point whatever edits exist are auto-submitted (§4). The prediction is the repo's final diff, graded by the standard SWE-bench fail-to-pass/pass-to-pass harness.

Results that matter

Two LMs (GPT-4 Turbo gpt-4-1106-preview, Claude 3 Opus), three settings: RAG (non-interactive), Shell-only (interactive, raw shell), SWE-agent (§4).

On cost: "compared to RAG on Lite, SWE-agent is 8-13x more costly but yields a 6.7-fold improved % Resolved rate" (§5) — interactivity is expensive per attempt, cheap per solved issue.

The two comparisons to internalize: SWE-agent vs RAG (12.47 vs 1.31, same LM) isolates interactivity; SWE-agent vs Shell-only (18.00 vs 11.00 on Lite, a "64% relative increase", §5) isolates the ACI itself. The ACI also transfers across models (Claude 3 Opus: 10.46%) and tasks — 87.7 pass@1 on HumanEvalFix Python (89.7 JS, 87.9 Java), state of the art at the time (Table 2).

Interface ablations, SWE-bench Lite % resolved (Table 3):

Two readings of this table repay attention. First, iterative search is worse than no search tools at all (12.0 vs 15.7): a bad interface actively harms, because agents exhaustively page through every result (§5.1, Fig. 5). Second, removing the edit command is the single largest drop (↓7.7): consolidated, guarded editing is the load-bearing component.

Behavior (§5.2): trajectories begin with reproduction and/or localization (all first actions are create or find_file/search_dir, Fig. 7); from turn 5 on, the loop is dominated by edit-then-execute (edit, python) cycles; 51.7% of all trajectories contain at least one failed edit. Resolved runs finish with a median cost of $1.21 and 12 steps vs a mean of $2.52 and 21 steps for failures — agents "succeed quickly and fail slowly", and 93.0% of resolved instances submit before exhausting the budget. Pass@k on Lite climbs steadily across 6 runs (Fig. 4) — an early sign that best-of-n sampling buys real resolve rate.

Where unresolved runs go wrong (GPT-4o-labeled failure modes, agreeing with the authors' hand labels on 87% of a validation set; §5.2, Fig. 8):

• 52.0% — incorrect (39.9%) or overly specific (12.1%) implementations: the agent "fixed" something, but not the issue, or only its literal reproduction.
• 23.4% — cascading failed edits.
• The remainder spreads across failing to find the relevant file or edit location, giving up prematurely, failing to reproduce, and running out of time/budget (Table 9 categories).

Limitations & how to defend the findings

• "12.47% still means 87.5% failure." Yes — and the failure analysis is part of the contribution: most failures are wrong solutions, not harness breakdowns (Fig. 8), and raising budget wouldn't help much since 93.0% of resolved instances submit before exhausting it while failures drag on (§5.2). The paper claims interface headroom, not solved SWE.
• "The ACI was tuned on GPT-4 — is this just overfitting an interface to one model?" The strongest skeptical read. Partial defense in-paper: the identical ACI gives Claude 3 Opus 10.46%/13.00% (Table 1), and HumanEvalFix transfers task-wise. But the authors are explicit that ACI design was driven by GPT-4 behavior on a dev split (§4), and open-source models (Llama 3, DeepSeek Coder) were dropped for subpar agent behavior (§4) — the ACI thesis is demonstrated on exactly two frontier LMs.
• "pass@1 hides variance." Addressed: per-instance resolution varies across the 6 runs but aggregate variance is low (Fig. 4, App. B.5); pass@k climbs steeply with k, foreshadowing today's best-of-n agent sampling.
• "Maybe any interactivity explains it, not the specific design." The Shell-only baseline exists precisely for this: interactivity alone gets 11.00% on Lite; the ACI adds 7 more points, and each component's removal measurably hurts (Table 3). The claim "ACIs tailored to LMs outperform UIs designed for humans" (§1) is backed by controlled deltas, not vibes.
• "Cost/practicality." $1.59 average per resolved-attempt instance with a $4 cap (§4, Table 1); the paper does not claim economic viability, and auto-submitting at budget slightly inflates apply rates.
• "Contamination?" §5 points to App. B.2: success is uncorrelated with issue age, "controlling for possible test pollution."

Connections

• SWE-bench: same core team, one paper later — the eval's authors showing the eval was bottlenecked by the harness, not only the models. Read the pair as problem-statement and first real solution: 3.79% (best RAG) → 12.47%.
• ReAct: the inner loop is ReAct's thought+action format (§3); SWE-agent's addition is everything around the loop — the action vocabulary and the observation shaping. ReAct fixed the reasoning pattern; SWE-agent fixed the environment.
• Why Claude-Code-style harnesses look the way they do: capped search output, line-windowed file reads, an edit primitive with immediate re-display, lint/validation guardrails on writes, history compaction of stale observations — every one of these is a Table-3-ablated component of this paper. The ACI vocabulary (view/search/edit/execute + submit) is the direct ancestor of modern coding-agent toolsets.
• Voyager: both papers discover that feedback shaping dominates raw model quality — Voyager's env-feedback/error/critique triad and SWE-agent's informative-concise-observation principle are the same lesson in different worlds; Voyager hand-built its ACI (Mineflayer APIs) without naming it.
• HCI lineage: the paper explicitly imports human-computer interaction methodology — study the user (here, an LM), instrument the interface, iterate (§2, §7) — proposing agent interface design as "systematic behavioral experimentation" on LMs.
• Computer-use agents: the ACI argument generalizes beyond code: wherever a GUI was designed for human perception, agents need a purpose-built layer (the paper's Fig. 2 analogy) — the intellectual seed of today's browser/computer-use tool designs.

Reading guide

1. §1 + Figs. 1–2 — the ACI concept and the headline claim; 5 minutes.
2. §2 — the four design principles; this is the reusable engineering checklist, read carefully.
3. §3 + Fig. 3 — the concrete command set and feedback formats; map each command back to a §2 principle as you read.
4. §5 + Table 1 — main results; internalize the RAG vs Shell-only vs SWE-agent ladder.
5. §5.1 + Table 3 + Figs. 5–6 — the interface ablations; Table 3 is the one table to study — it is the empirical content of the ACI thesis, one deletion at a time.
6. §5.2 + Figs. 7–8 — agent behavior patterns (localize → edit/execute loops; succeed fast, fail slow); the section modern harness builders quote most.
7. Skimmable: §6 related work, §7 discussion. Appendices A (implementation), B.3 (trajectory statistics) if you're building your own harness; App. C shows full prompt/demonstration text.