"Thinks like a frontier model"
A well-tuned 9B model handles summarization, boilerplate code, and drafting well. On multi-step reasoning, frontier models still lead by a wide margin on benchmarks like GPQA. Right tool, right job.
The Fit Calculator
The 3D tower is your unified memory. Purple blocks = model weights, blue = KV cache (context), grey = OS + apps. Drag to orbit; change settings and watch it fill.
Drag to rotate · wheel/pinch to zoom
5.6 GBModel file
7.4 GBRAM needed
1.1 GBKV cache
~18 t/sSpeed est.
Fits comfortably
The one equation that matters
A model's download size is roughly its parameter count times bytes per weight. Quantization shrinks each weight from 16 bits down to 4–8 bits with modest quality loss:
file_size ≈ params × bits_per_weight / 8
9B × 5 bits / 8 ≈ 5.6 GB ← why "5.6 GB 9B model" is a Q5 quant
RAM_needed ≈ file_size × 1.15 + KV_cache(context)
9B × 5 bits / 8 ≈ 5.6 GB ← why "5.6 GB 9B model" is a Q5 quant
RAM_needed ≈ file_size × 1.15 + KV_cache(context)
That is why the same 9B model ships as a 18 GB FP16 file, a 9.5 GB Q8, or a 5.6 GB Q5 — identical architecture, different precision per weight.
What you trade at each level
| Precision | Bits/weight | 9B model size | Typical quality | Use when |
|---|---|---|---|---|
| FP16 | 16 | ~18 GB | Reference quality | Benchmarking, fine-tuning |
| Q8 | 8 | ~9.6 GB | Nearly indistinguishable | You have RAM to spare |
| Q5 | ~5 | ~5.6 GB | Slight loss on hard reasoning | The everyday sweet spot |
| Q4 | ~4 | ~4.8 GB | Noticeable on math/code edge cases | 8 GB machines, long contexts |
Reality-checking the hype
"Nothing leaves your machine"
True — and genuinely valuable. Local inference means medical notes, contracts, and proprietary code never touch a third-party server. This is the strongest real argument for local models.
"Free"
Free to download, but you pay in electricity, RAM pressure, and speed. On a base 16 GB laptop, a Q5 9B model generates roughly 10–25 tokens/sec — fine for chat, slow for bulk processing.
Verify before you install
Model names in viral posts are often wrong or invented. Check the actual model card on Hugging Face or Ollama's library: parameter count, license, quantization, benchmark scores.