Why the denominator shrinks
A length-stopped answer is not clean evidence of model quality. The gate preserves it, names the item, and removes the entire pair from both accuracy denominators so baseline and candidate remain comparable.
Freeze the evaluator. Measure the run.
Normalize mixed answers, withhold truncation, and compare only paired evidence. The bundled fixture is synthetic and local; no model is trained or queried.
Paired evaluation CSV
8 REQUIRED COLUMNSProtocol fields: run, item_id, task, answer_format, expected, answer, finish_reason, max_tokens. Lists are order-independent; object keys are canonicalized.
No comparison yetRun the frozen sample to make the measurable subset and rerun queue explicit.
SCHEMA FROZENPAIRED BY ITEM IDTRUNCATION WITHHELDEXACT NORMALIZED MATCH
Decision
—
—
—VALID ROWS
—COMPARABLE PAIRS
—BASELINE ACCURACY
—CANDIDATE ACCURACY
| Item | Format | Baseline | Candidate | Evidence decision |
|---|
—
Counterexample: the headline alone
Reporting 75% candidate accuracy across all four items would mix three completed answers with one truncated response. The paired gate reports 100.0% on three comparable items and keeps the +66.7 pp gain provisional until q3 is rerun.