Coding-agent systems laboratory
The model writes code. The scaffold shapes the run.
Change planning, tools, retries, tests, and context strategy. See how those assumptions alter a simulation of token burn, elapsed time, and task success.
1Planshape work
2Actuse tools
3Testobserve
4Recoverretry or stop
Simulated outcome
Balanced scaffold
Token-burn distribution
Frequency across simulated runs
SolvedUnsolved
One execution, step by step
A representative seeded run
1 / 1
Scenario comparison
Find the efficient frontier
Points farther up succeed more often; points farther left use fewer tokens. A point is Pareto-efficient when no other saved scenario is better on both measures.
Interpret the model
Read the tradeoffs, not just the score
A scaffold does not improve the underlying model in this lab. It changes how often the model gets structured opportunities to inspect, recover, and stop.
Spend before acting
Planning adds an up-front token and time cost. The simulator assumes moderate planning can improve each attempt, with diminishing returns at high depth.
Catch plausible failures
Tests add work but can reject an incorrect attempt. When a failure is detected, a retry can use that evidence rather than repeat the same path.
Choose what persists
Full history compounds token usage. Summaries preserve some evidence with less growth. Fresh retries stay cheap but discard useful observations.
Transparent assumptions
What the simulator computes
Each trial draws success from a seeded pseudo-random sequence. Attempt probability starts with the selected base rate, then receives bounded adjustments for planning, tests, and targeted recovery. Token totals combine reasoning, planning, tools, verification, and context growth. Time combines token throughput and tool latency.
Important: these coefficients are illustrative and editable only through the controls. They are not measured values from the Scaffold Effect paper.
- Attempt probability
- base + planning lift + verification lift + retry evidence
- Attempt tokens
- reasoning + plan + tool calls + tests, multiplied by context growth
- Task success
- fraction of trials with at least one accepted successful attempt
- Cost / success
- all simulated token cost divided by successful trials
Knowledge check