Hardware Profile

Preset (fully editable)

System RAM (GB)

GPU / unified memory (GB)

Memory bandwidth (GB/s)

Compute class (1 low – 3 high)

Storage

Sustained power (W)

Purchase price ($)

Lifetime (months)

Model & Workload

Task preset

Parameters (B)

Quantization bits: 4

Context (tokens): 4096

Batch size

Concurrent users/agents

Target tok/s per user

Memory Map (weights / KV / runtime / OS / margin)

Results & Fit

Checking…

Bandwidth-limited throughput

Illustrative: decode speed ≈ bandwidth ÷ weight bytes read per token, split across concurrent streams.

Scenario Comparison

Duplicate the current configuration to compare setups side by side.

Local vs Cloud Comparison

Monthly cloud/API spend ($)

Private-data sensitivity

Offline required

Utilization (hrs/day)

Electricity ($/kWh)

The quality gap cannot be reduced to one number: frontier cloud models generally lead on hard reasoning and breadth, while a well-chosen local model can be excellent for private document Q&A, extraction, drafting, and offline work. Local can win on privacy, availability, customization, and learning even when cloud is cheaper or more capable.

Hybrid Workload Planner

Rules: private or offline → local; quality-critical and not private → cloud; bursty and not private → cloud; latency-sensitive small tasks → local; mixed signals → hybrid (local first, cloud escalation).

Learning Chapters

Parameters & quantization

Parameter count sets the raw memory footprint: bytes ≈ params × bits ÷ 8. Quantizing from 16-bit to 4-bit cuts weight memory ~4× with modest quality loss for many tasks; below 3-bit, degradation grows quickly.

KV cache & context length

Every token in context stores key/value activations per layer. KV memory grows linearly with context, batch, and concurrency, and can exceed the weights themselves at long contexts.

Memory bandwidth & tokens/sec

Decoding reads most weights once per generated token, so tokens/sec is often bandwidth ÷ weight size, not raw FLOPS. This is why unified-memory machines with high bandwidth punch above their compute class.

Unified vs discrete memory

Discrete GPUs have fast but small VRAM; unified memory offers larger pools at moderate bandwidth. A model that spills out of VRAM to system RAM typically slows drastically.

Local privacy & hybrid routing

Local inference keeps data on your machine, works offline, and is fully customizable. Hybrid routing sends private or latency-sensitive work locally and escalates hard, quality-critical requests to a cloud model.

Why model quality varies by task

Small models can rival large ones on narrow tasks like extraction or retrieval-grounded Q&A, while lagging on multi-step reasoning and broad knowledge. Match model size to the task rather than assuming one answer.

Knowledge Check

Assumptions & Formulas

weights_GB = params_B × bits/8 × 1.1 (10% overhead)
kv_GB = params_B × 0.00012 × context × batch × concurrency ÷ 1.024 (≈0.12 MB/token per 1B params at 16-bit KV)
tok/s ≈ bandwidth_GBs ÷ weights_GB × efficiency(0.5/0.65/0.8 by compute class) ÷ concurrency
TTFT ≈ context ÷ (prefill = 300 × params-scaled compute factor)
electricity $/mo = W ÷ 1000 × hrs/day × 30 × $/kWh; local monthly cost = price ÷ lifetime + electricity. Breakeven month = price ÷ (cloud spend − electricity) when positive. OS reserve 4 GB (discrete GPU: 1 GB of VRAM), runtime 1.5 GB, safety margin 8% of pool. All outputs are illustrative estimates; this page never inspects your machine or runs benchmarks.