Agent-as-a-Judge vs human evaluation
Human review is the gold standard for judgment quality but it does not scale: a careful reviewer covers 20-50 trajectories a day, gets fatigued, and drifts. An LLM judge scores thousands per hour at near-zero marginal cost, which is what makes a flywheel possible at all.
- Calibrate first: have humans label a few hundred trajectories, then measure judge-human agreement before trusting the judge.
- Rubrics beat vibes: judges given explicit criteria (task completed? tool errors? unsafe actions?) agree with humans far more than open-ended scoring.
- Known biases: LLM judges favor longer answers, their own model family, and the first option shown; mitigate with position-swapping and length normalization.
- Escalation: route low-confidence or disagreement cases to humans; automate the easy 90%.
Consistency metrics that matter
Before an automated judge replaces human eval, teams verify it is consistent with humans and with itself.
- Agreement rate: fraction of items where judge and human give the same verdict. Useful but inflated when one class dominates.
- Cohen's kappa:
k = (p_o - p_e)/(1 - p_e), agreement corrected for chance. Above ~0.7 is generally considered strong for eval work. - Self-consistency: ask the judge the same question N times; the vote entropy reveals flaky criteria.
- Pairwise win-rate correlation: when ranking two agent versions, does the judge's preference ordering match human preference ordering (Spearman rho)?
The four stations, in practice
- Trajectory observability: instrument every run with spans: each LLM call, tool call, retry, and token count. Without traces there is nothing to learn from; failures are anecdotes instead of data.
- Trace2Dataset: convert raw traces into eval cases automatically: input, expected behavior, and the failure label. Production failures become tomorrow's regression tests.
- Agent-as-a-Judge: a judging agent inspects whole trajectories, not just final answers: did it pick the right tool, recover from errors, avoid loops?
- Memory libraries: distilled lessons (working plans, tool quirks, judged exemplars) are stored and injected into future runs, so the agent improves without retraining.
Why flywheels beat one-off evals
A static benchmark measures an agent once; a flywheel improves it continuously. The compounding comes from the loop, not any single stage.
- Each cycle both measures quality and generates the data that raises it.
- Eval coverage grows automatically as production surfaces new failure modes.
- Faster cycles mean regressions are caught in hours, not release cycles.
- Caveat: a biased judge automates its bias at scale. Periodic human audits of the judge are the brake that keeps the flywheel honest.
Worked example: what one extra cycle per week is worth
Suppose each improvement cycle fixes failure modes worth a 2% absolute gain in task success rate, with diminishing returns. At 1 cycle/week (fully manual judging) an agent at 70% success reaches roughly 78% in a month. At 8 cycles/week (automated judging with human audits) the same team runs 32 cycles and plateaus near its data-quality ceiling in the same month, then spends the rest of the quarter expanding coverage instead of waiting on reviews.
The arithmetic behind the slider above:
stage_hours(a) = manual + (automated - manual) * awhereais the automation fraction from the slider.cycle_hours = sum of the four stage hours; judging dominates at low automation (24h manual vs 1.5h automated).cycles_per_week = 40 / cycle_hours, capped in practice by how fast you can ship agent changes.- Fully manual: 52h per cycle, about 0.77 cycles/week. Fully automated: 3.5h per cycle, about 11.4 cycles/week, a ~15x speedup.
Two honest caveats. First, automation quality gates the whole loop: a judge with kappa 0.5 against humans produces fast noise, not fast learning. Second, the flywheel measures what the judge can see; silent failure modes (subtle factual errors, slow degradation of tone) still need scheduled human deep-dives. Mature teams typically settle around 80-90% automation with a standing 5-10% human audit sample.