From toy functions to real repositories
What do software engineers really do?
That was the question behind SWE-bench, conceived in the summer of 2023 — back when HumanEval (164 standalone puzzle functions) was the standard for judging code models. The answer: engineers don't write functions from scratch in a vacuum. They read big messy codebases, understand bug reports, and change the right 20 lines out of 200,000. Pick a benchmark below and see the difference in 3D.
Drag to rotate · panels = code the model must navigate · tiles = tests
SWE-bench (2023)
A short history of asking machines to code
HumanEval — the function era
Released with OpenAI's Codex paper: 164 hand-written Python problems ("check if any two numbers in a list are closer than a threshold"). Self-contained, no repository context, graded by unit tests. Perfect for its moment — and quickly saturated, memorized, and gamed.
SWE-bench — the repository era
Built at Princeton by scraping 2,294 resolved issues from 12 popular Python repos (Django, scikit-learn, sympy, matplotlib…). The model gets the issue text and the full pre-fix codebase, and must produce a patch. Grading uses fail-to-pass tests taken from the real human fix — the tests that failed before the maintainer's patch and passed after. Early result: the best model resolved under 2% of issues.
SWE-bench Verified — the trust era
Human annotators found some original tasks were underspecified or had unfair tests, so a 500-task human-validated subset was released. It became the industry's headline coding number — and resolve rates climbed from single digits toward and past 70% as agent scaffolding (editing tools, test running, retries) matured.
Beyond the patch
Descendants like SWE-bench Multimodal and Multilingual, plus live/rotating variants, chase the next gap: contamination resistance, non-Python stacks, and tasks closer to full tickets than single patches. The core lesson stands — measure the job, not a proxy for the job.
Worked example — one SWE-bench task, end to end
Input. Issue from the sympy repo: Rational('0.5', '100') returns 1/100100 instead of 1/200 — a real parsing bug report, plus a snapshot of the repo at the commit before the fix (~700k lines).
Localization. The agent must find where string arguments are parsed — a handful of relevant lines inside sympy/core/numbers.py. This search step is where most early models failed; nothing in HumanEval ever required finding the code first.
Patch. Produce a diff that parses each string argument independently before forming the fraction. A correct fix is ~6 changed lines.
Grading. Harness applies the patch and runs the repo's tests: the fail-to-pass tests (written for the human fix) must now pass, and all previously passing tests must still pass. No LLM judge, no style points — the repo's own test suite decides.
Score. A task counts only if fully resolved. Resolve rate = resolved / total. That single number is what you see quoted whenever a new model claims a coding breakthrough.