No API bill. No data leaving your laptop. The tweet-sized promise is real — but it runs on very concrete math: bytes per parameter, memory bandwidth, and quantization. Here's how local inference actually works.
Pick a model size and quantization. The 3D memory tower shows the model's weight blocks stacking into a GPU's VRAM (the translucent frame = your memory budget). Drag to rotate.
Drag with mouse or finger to orbit the memory tower
A model is just billions of numbers (weights). Storage depends on how many bytes each number takes:
Worked example — Llama-class 8B: at FP16 (2 bytes) → 8 × 2 = 16 GB. At 4-bit (~0.56 bytes incl. scales) → ~4.5 GB. Same brain, quarter the memory.
Generating one token means reading every weight from memory once. Decode is memory-bandwidth-bound, so:
Worked example: RTX 4090 has ~1,008 GB/s. An 8B model at Q4 is ~4.5 GB → theoretical ceiling ≈ 224 tok/s; real-world ~60–70% of that. That's why smaller quants feel faster, not just lighter.
Quantization stores each weight with fewer bits, grouped into blocks with a shared scale factor. The GGUF format (used by llama.cpp and Ollama) packages quantized weights plus tokenizer and metadata in one portable file.
| Format | Bits/weight | ~File size | Quality impact |
|---|---|---|---|
| FP16 | 16 | 16.0 GB | Reference quality |
| Q8_0 | ~8.5 | 8.5 GB | Virtually lossless |
| Q6_K | ~6.6 | 6.6 GB | Near-lossless |
| Q4_K_M | ~4.8 | 4.9 GB | Small, usually acceptable loss — the community default |
| Q2_K | ~2.6 | 3.2 GB | Noticeable degradation; last resort |
Rule of thumb: a bigger model at 4-bit usually beats a smaller model at 8-bit. 70B-Q4 > 8B-FP16 on most tasks — if you have ~40 GB of memory to hold it.
The C/C++ inference engine underneath almost everything. Runs GGUF on CPU, CUDA, Metal, Vulkan. Maximum control, CLI-first. You compile flags; it rewards you with speed.
llama.cpp wrapped in a one-line experience: ollama run llama3.1:8b. Pulls models, manages versions, exposes an OpenAI-compatible local API on port 11434 — so existing agent code can point at localhost.
A desktop GUI for browsing, downloading, and chatting with GGUF models. Shows you VRAM fit estimates before download. Friendliest on-ramp for non-terminal people.
| Dimension | Local (Ollama / llama.cpp) | Cloud API |
|---|---|---|
| Marginal cost | $0/token (you pay electricity + hardware) | Per-token billing, scales with usage |
| Privacy | Data never leaves the machine | Data transits provider servers |
| Peak quality | Capped by what fits in your RAM | Frontier models, hundreds of billions of params |
| Latency | No network hop; speed set by your bandwidth | Network + queue, but huge batched throughput |
| Ops burden | You manage updates, prompts, context limits | Provider handles everything |
The sane strategy for an "AI agent that runs free": local model for high-volume, private, routine steps; cloud call only for the hard reasoning. That's not a hack — it's how production systems cut bills by 10× or more.