Skip to content
Agent Friendly Code

Coding agent benchmarks: what SWE-bench and others measure

Coding agent benchmarks run a model, usually inside an agent harness, on a fixed set of tasks and count how many it solves: SWE-bench and its successors use real GitHub issues, Terminal-Bench uses terminal tasks, Aider Polyglot uses programming exercises. They rank models and agents, not how well your repo is set up for one.

What do coding agent benchmarks measure?

Each fixes the tasks and repos, then varies the model or agent. The score is the share of tasks solved, usually checked by tests written for each task. That makes results comparable across models, and means every number describes the benchmark's repos, not yours.

How do SWE-bench, Terminal-Bench and the others compare?

Popular coding agent benchmarks: size, languages and what each scores
BenchmarkTasksLanguagesWhat it tests
SWE-bench Verified500, a human-validated subset of SWE-bench's 2,294 issues from 12 reposPythonResolving a real GitHub issue; the fix passes when the repo's hidden tests pass
SWE-Bench Pro (Scale AI)1,865 across 41 professional repos: 731 public, 276 commercial from 18 startup codebases, 858 held outResults reported for Go, Python, JavaScript and TypeScriptLong-horizon, multi-file changes (bugs, features, security updates); resolved when the new tests pass and the old ones still do. Built to resist contamination
Terminal-Bench 2.089 hard tasks in terminal environmentsNot language-specificCompleting a task in its own terminal environment, verified by tests written for it
Aider Polyglot225 hard Exercism exercisesC++, Go, Java, JavaScript, Python, RustWhether an LLM can follow instructions and edit code to solve each exercise without human help
Multi-SWE-bench1,632 instances checked by 68 expert annotators; flash (300) and mini (400) variantsJava, TypeScript, JavaScript, Go, Rust, C, C++Issue resolution, as in SWE-bench, beyond Python

tbench.ai's leaderboard shows a version later than 2.0, so check which one a quoted score used.

SWE-bench vs Terminal-Bench: what's the difference?

SWE-bench hands the agent a repository and an issue and checks the patch with tests. Terminal-Bench gives it a terminal environment and a task drawn from real workflows, checked by tests written for that task. One measures reading and fixing an existing codebase; the other, getting work done through the command line. An agent can do well on one and poorly on the other.

Which languages does SWE-bench cover?

Only Python: the original SWE-bench, and so Verified, comes from 12 popular Python repositories. For other languages see Multi-SWE-bench (seven languages), SWE-Bench Pro (results for Go, Python, JavaScript and TypeScript) or Aider Polyglot (six languages, on exercises rather than repos). A model's Python issue-fixing score says little about how it handles your Rust or Java.

How should I read an AI coding agent leaderboard?

  • Check what was run: a bare model, or a model inside a particular agent. The same model scores differently in different harnesses.
  • Check the version and subset: SWE-bench Verified is not SWE-bench, and Terminal-Bench 2.0 is not its later versions.
  • Check the languages against yours.
  • Treat small gaps as noise; a few tasks either way can reorder the top.
  • Leaders change often: read the benchmark's own site, not a quoted number.

What do benchmarks leave out?

Your repo. A benchmark holds the codebase fixed and asks which agent is best. In your work the agent is usually fixed and the codebase varies: an AGENTS.md, tests the agent can run, CI, a linter, a reproducible dev setup. Those decide how much of a model's benchmark score you get.

Our leaderboard asks that second question. It scores 402 public repos, once per agent for 9 agents, on file checks weighted by what each agent's docs say it reads. No agent is run, so it measures whether the scaffolding agents rely on is there, not outcomes. The methodology page lists every check and weight.

How do a benchmark score and a repo score fit together?

  1. Use benchmarks to shortlist models and agents by the closest task type and language to your work.
  2. Use a repo score to see what your codebase is missing for the agent you picked: an instructions file it actually reads, runnable tests, setup.
  3. Fix the repo side first if it's weak: it helps every agent, and doesn't change when the leaderboard does.

Questions people ask

What is SWE-bench Verified?
A human-validated subset of 500 SWE-bench tasks: real GitHub issues from Python repositories, each checked by the repo's tests.
Is SWE-bench only Python?
Yes, SWE-bench and SWE-bench Verified are Python only. Multi-SWE-bench covers seven other languages, and SWE-Bench Pro reports Go, JavaScript and TypeScript as well as Python.
Who is top of the coding agent leaderboards?
It changes too often to state here. Each benchmark's site (swebench.com, Scale's SWE-Bench Pro page, tbench.ai, Aider's leaderboard) has the current table.
Does a higher benchmark score mean an agent will do better in my repo?
Only if your tasks resemble the benchmark's, and your repo gives the agent what it needs: instructions, tests it can run, a working setup.
Is Agent Friendly Code a benchmark?
No. It ranks repos, not agents, by checking files; no agent is run. It complements benchmarks.

From the docs: swebench.com/verified.html (new tab) · swebench.com/original.html (new tab) · labs.scale.com/swe_bench_pro_public (new tab) · arxiv.org/2601.11868 (new tab) · tbench.ai/announcement-2-0 (new tab) · aider.chat/leaderboards (new tab) · github.com/multi-swe-bench (new tab)

How we score repos · The leaderboard · Score your repo · Every check we run · Check your AGENTS.md · What each agent reads