LLM-as-judge works — until the bill and the bias arrive
Most teams validate agent outputs by asking a frontier model "was this good?" It's a fine way to start and a terrible way to scale. Three problems stack up fast: cost (a frontier call on every turn), latency (evals slower than the agent), and bias drift (judges prefer their own family's style). Model your own eval stack below.
The judge strategy calculator
Latency
Human agreement
Your workload
The three problems, precisely
Cost compounds
20k judged turns/day × 1.5k tokens × frontier pricing (~$5–15/M input, more for output) ≈ $300–900/day — often more than the agent itself. Evals should be cheaper than the thing they evaluate.
Latency blocks CI
2–6 s per frontier judgment means a 5k-case eval suite takes hours sequentially. Teams respond by sampling 2%, which quietly turns "we eval everything" into "we spot-check."
Bias and drift
Documented judge biases: position bias (prefers the first answer shown), verbosity bias (longer = better), self-preference (rates its own model family higher), and silent drift when the provider updates the judge model under you.
The hybrid eval stack that scales
Programmatic checks first: JSON validity, schema, required citations, regex bans, tool-call success. Free, instant, catches 30–50% of failures. Never pay a model to check what a regex can.
A fine-tuned 7–8B judge (trained on ~2–5k human-labeled examples from YOUR domain) handles the volume. Typical result: 85–93% agreement with humans at 1/20th the cost and ~300 ms latency.
Reserve the frontier judge for the 5–10% of cases the small judge flags as uncertain, plus a random 1% audit stream to detect small-judge drift.
Weekly human review of ~100 sampled judgments. Humans calibrate the judges; judges never calibrate themselves. Track judge-human agreement over time — if it falls, retrain Tier 2.