The math is simpler than you think
01 Weights dominate
Every parameter is stored as a number. At FP16 that is 2 bytes per parameter, so a 9B model needs about 9 × 2 = 18 GB just for weights. Quantize to 4-bit (~0.55 bytes with overhead) and the same model shrinks to roughly 9 × 0.55 ≈ 5 GB — a 3.6× reduction with only a small quality dip.
02 KV cache grows with context
While generating, the model caches attention keys and values for every token in context. For a typical 9B architecture that costs roughly 0.15 MB per token. An 8K context adds ~1.2 GB; a 128K context can add more than the weights themselves. Long context is a memory feature, not a free lunch.
03 Why "self-fixing loops" work locally
Agentic coding models run plan → write → test → fix cycles. Locally, each retry costs zero API dollars — only electricity and time. A loop that takes 40 attempts at $0.02 per call would cost $0.80 in the cloud; locally it costs nothing, which changes how aggressively you can let an agent iterate.
04 Tokens per second reality
Speed scales with memory bandwidth. A laptop with 100 GB/s bandwidth running a 5 GB (Q4) model tops out near 100 ÷ 5 = 20 tokens/sec. The same laptop on an 18 GB FP16 model crawls at ~5.5 tokens/sec. Quantization is a speed upgrade, not just a size trick.
| Model | FP16 | Q8 | Q4 | Runs on 16 GB laptop? |
|---|---|---|---|---|
| 3B | 6 GB | 3 GB | 1.7 GB | Yes, easily |
| 9B | 18 GB | 9 GB | 5 GB | Yes at Q8/Q4 |
| 34B | 68 GB | 34 GB | 19 GB | No (needs ~24 GB) |
| 70B | 140 GB | 70 GB | 39 GB | No (workstation class) |
Quick check
A 12B model at Q4 (~0.55 bytes/param) with a small context needs roughly how much memory?