The Scaffold Effect Simulator
What if the harness matters as much as the model itself — both for how many tokens you burn and how often you actually solve the task? Design a harness below, run 1,000 Monte Carlo trials in your browser, and watch the cost/success frontier move.
Step 1
Build Your Harness
Every knob changes both expected token spend and probability of a verified solve.
Each pass catches 60% of remaining bad solutions but costs tokens.
Step 2
Live Simulation — 1,000 Trials
0%Solve rate
0Median tokens / trial
—Median cost / solved task
Step 3
The Agent Loop, Weighted by Spend
Node size pulses in proportion to expected token spend per stage under your current settings.
Step 4
Compare Harnesses
Pin configs to see the tradeoff. X: median tokens per trial. Y: solve rate. The Pareto frontier is drawn in amber.
Worked example
Cheap & Greedy35% solve, 1 retry, fresh context, 2 tool calls, 0 verification.
Low median spend, but with zero verification some accepted answers are silently wrong — the real solve rate is lower than it feels.
Worked example
Verify Everything35% solve, 6 retries, summarized context, 6 tool calls, 3 verification passes.
Much higher verified solve rate, but each attempt costs far more — the cost per solved task can still be favorable because fewer wrong answers slip through.