AI Evals Explained: Runs, Judges & Thresholds

Runs per case 3

Pass threshold 70%

Task difficulty medium

Drag to rotate · 12 test cases flow through the judge gate; green = pass, coral = fail

—

Cases passed

—

Suite pass rate

—

vs threshold

—

Est. cost

Why one run is never enough

LLMs are stochastic: the same prompt can pass a test today and fail it tomorrow. A single run tells you almost nothing. Evals fix this with repeated runs:

pass rate — fraction of runs that succeed (7/10 = 70%).
pass@k — did any of k attempts succeed? Useful when a human picks the best output.
variance — if a case passes 3/10 runs, the feature isn't "working," it's gambling.

Worked example

A case with true 60% per-run success: 1 run misleads you 40% of the time. Across 10 runs, the chance of seeing 0 passes is 0.4¹⁰ ≈ 0.01% — you now measure reliability, not luck.

The vocabulary, decoded

Judge model

A second LLM grades the output against a rubric ("did the agent fix the bug without breaking tests?"). Cheaper and faster than human graders; you audit a sample to keep it honest.

Threshold

The minimum pass rate to ship. CI can fail the build if the suite scores below it — exactly like code coverage gates, but for model behavior.

Ablation

Re-run the suite with one ingredient removed (a tool, a prompt section, a scaffold) to measure what that ingredient is actually worth.

Cost caps

Flags like --max-cost-usd keep a 10-run × 50-case suite from surprising you with a large API bill.

A minimal eval loop you can copy

for case in suite: # e.g. 12 coding tasks results = [] for i in range(runs): # --runs 3 out = agent.run(case.prompt) score = judge(out, case.rubric) # --judge-model results.append(score) case.pass_rate = mean(results) suite_rate = mean(c.pass_rate for c in suite) exit(0 if suite_rate >= threshold else 1) # --threshold 0.7

That's the whole idea. Everything else — tags, ablations, scaffolds, output dirs — is tooling around this loop so you can compare configurations honestly instead of vibe-checking your agent.