More capability per FLOP — the allocation lab

The question, taken seriously

“For those of us actually training or running models locally: what if better pre-training methods could give us more capability per FLOP? the entire pre-training paradigm we’ve been scaling for years isn’t general enough. The Bitter Lesson might not be bitter enough anymore.”

“Capability per FLOP” sounds like a slogan, but it is a measurable quantity, and the field has a published instrument for it: the parametric loss fit from Hoffmann et al. (2022), the Chinchilla paper. This lab puts that instrument under your hands. Nothing on this page is simulated data or decoration — every number is the fit, evaluated live.

The model under the terrain

N is parameters, D is training tokens, C is total training FLOPs, and m is an effective-data multiplier for a better pre-training method (m = 1 is the published fit). The first term is the capacity wall: small models cannot represent what the data contains. The second is the data wall: big models starved of tokens never learn what they could hold. A fixed budget C is not a knob — it is a constraint line, and training well means choosing where to stand on it.

The frontier — the bottom of the U along the rope — has a closed form: N_opt = 1.3447·(mC/6)^0.4516. At the optimum the two walls push back equally (α·capacity term = β·data term), which is why the frontier is a ridge-bottom river on the terrain above.

The Bitter Lesson, priced instead of argued

Rich Sutton’s essay says general methods that ride growing compute beat hand-built knowledge. The post asks whether that lesson is “bitter enough anymore.” The fit gives a sharper answer than either slogan: at the optimum, reducible loss depends only on the product m·C. A method that makes each token teach twice as much is indistinguishable from doubling your FLOPs — you can verify it in Step 3, where m = 3 at 1e21 FLOPs lands exactly on the 3e21 frontier (2.2297 nats both ways).

So better pre-training methods don’t repeal the Bitter Lesson; they trade in its own currency. For people training locally, that is the actionable version of “more capability per FLOP”: a 3× data-quality method is a rented cluster you didn’t pay for. The catch — and the honest part — is that m is a hypothesis you must measure for your own pipeline, which is exactly what small local ablations are good at.

Assumptions and limits (read before believing)

• The constants (E=1.69, A=406.4, B=410.7, α=0.34, β=0.28) are the approach-3 parametric fit from Hoffmann et al. 2022, fitted on runs up to ~70B params and ~500B tokens. Budgets here stay near that support, but treat far extrapolation with suspicion.
• C ≈ 6ND ignores attention and embedding overhead; it is the standard accounting, good to tens of percent.
• Different fitting approaches in the same paper give compute-optimal ratios from ~20 to ~70 tokens per parameter; the U-shape and the frontier logic are robust, the exact ratio is not gospel. Later fits (e.g. Epoch AI’s reanalysis) shift the constants without changing the lesson.
• Cross-entropy loss is not downstream capability; it is the best-behaved proxy we can fit. Inference cost is deliberately out of scope — overtraining smaller models past the frontier is rational when you serve the model often.
• The method multiplier m is a learner-set hypothesis (“what if my data taught m× more per token”), not a measured promise. The lab prices the hypothesis; your ablations must earn it.
• GPU-time equivalences assume dense BF16 throughput at 40% utilization (4090 ≈ 33 TFLOP/s effective, A100 ≈ 125 TFLOP/s effective).