Does Your Coding Agent Upgrade Earn Its Cost?

Use SWE-bench Verified to inform patch tasks and Terminal-Bench to inform shell tasks. Qualify cline-bench from the release’s actual tasks and graders, then price the upgrade using your own paired results and workload mix.

Keep patch acceptance distinct from terminal completion

Use the SWE-bench issue-to-patch format for work where a repository change is the deliverable. For a local fixture, pin the checkout and hand both agents the same bug report. Save each diff and grade it against a regression check plus the relevant existing tests. A patch that passes the new regression but breaks an existing contract earns a failure. Give each task one paired outcome for the upgrade decision, even if the agent generates several candidate patches internally. Report gains and regressions alongside the net change. In an illustrative ten-task trial, both an agent that gains three and loses two and one that gains one and loses zero improve by ten percentage points. The first introduces two regressions worth inspecting before rollout.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models Resolve Real-world Github Issues? · GitHub

Grade terminal tasks by observable end state

Use Terminal-Bench's terminal environment framing when the deliverable is an artifact or working environment. A local fixture can ask the agent to rebuild a package from a clean image, then grade the resulting artifact by running a deterministic smoke check against it. Fix the image, command permissions, deadline, and allowed resources across both configurations. For cline-bench, examine the chosen release's tasks and grading before assigning it weight. For example, place an engineering-session task in your shell bucket only if its grader checks the end state you care about. Weight patch and shell results by incoming workload volume. Report the sample size for each bucket so a shell pass-rate advantage from a handful of tasks stays visible in the decision.

GitHub - harbor-framework/terminal-bench-1: A benchmark for LLMs on complicated tasks in the terminal · GitHub

Match the benchmark to the deliverable

The core SWE-bench task is to turn a codebase and an issue into a patch that resolves the problem. For a queue of narrowly scoped bug fixes, look at SWE-bench Verified and confirm the exact task IDs behind each reported score. For changes spanning several components, examine SWE-bench Pro and inspect representative tasks for the kinds of coordination your team needs. Compare results within the same release and task set.

For cline-bench, verify the provenance of the engineering-session tasks, what context the agent receives, and how the grader decides success. The available excerpt establishes the original SWE-bench task contract. Apply a common check before relying on any of the repository suites: take one ordinary patch from your backlog and see whether the suite expects a comparable artifact and tests comparable failure modes.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models Resolve Real-world Github Issues? · GitHub

Record the harness with the score

Use Terminal-Bench when the deliverable is a completed shell workflow, such as producing an artifact that passes a validator. The repository's beta example specifies the agent, model, dataset name, and dataset version separately. For Terminal-Bench 2.0, verify the dataset identifier and evaluator in that release's own documentation before comparing results. The beta example establishes how one historical run was configured.

To test a model upgrade, fix the harness, task IDs, starting images, permissions, time budget, and retry policy. To compare complete agent products, run each in its intended configuration and label the result with both the harness and model. Save the grader result, exit status, and terminal log for every attempt. A pass requires the expected artifact or final state to satisfy the grader, even when the agent's summary sounds convincing.

GitHub - harbor-framework/terminal-bench-1: A benchmark for LLMs on complicated tasks in the terminal · GitHub

Count what changed on paired tasks

In an illustrative upgrade trial, compare two model versions in the same harness on 12 held-out issue repairs from your repository. Give each version one 20-minute attempt per issue in a fresh environment. Use the issue-to-patch format from SWE-bench and define a local pass as a patch that passes both the issue-specific check and the regression suite. Suppose you require at least two additional passes and zero lost passes to approve the upgrade.

If both versions pass six tasks, only the candidate passes three, only the incumbent passes one, and both fail two, the candidate scores nine against seven. Hold the upgrade under the stated rule. It met the improvement target and lost one previously working task. Inspect that task's diff and logs before deciding whether to change the configuration or keep the incumbent.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models Resolve Real-world Github Issues? · GitHub

Separate startup failures from bad patches

SWE-bench uses Docker for reproducible evaluations. Before a scored batch, run the evaluator on a known-good patch and verify that the issue check fails on the base revision and passes with the patch applied. Record setup failures, missing patches, and patches that fail tests as distinct outcomes. Keep the first-attempt completion count alongside any separate tally of patch correctness.

Suppose two candidate runs stop before the repository opens because the evaluation image cannot start. Restore the intended image and rerun those tasks under the same limits. If both configurations can then start, you have evidence that the initial results were affected by setup. If only one harness still fails to start in your intended environment, count those attempts against its operational completion rate. Give readers the attempted and gradable counts so they can interpret both measures.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models Resolve Real-world Github Issues? · GitHub

Sources