Local LLM Hardware & Token Speed Simulator

Model & Quantization 8B Q4_K_M

Model Parameters 8 Billion

Quantization Precision

FP16 (16-bit)

Q8_0 (8-bit)

Q4_K_M (4.5b)

Q2_K (2.5b)

Active Context Window 4,096 tokens

Batch Size / Streams 1 request

Hardware System Specs

Dedicated GPU VRAM 24 GB

System RAM (CPU Offload) 64 GB

Memory Bandwidth 800 GB/s

Economics & Usage

Local Hardware Investment $1,600

Cloud AI API Subscription $20 / mo

Daily Prompt Requests 500 req/day

Deployment Feasibility

100% VRAM

Fully fits in high-speed GPU VRAM

Gen Throughput Limit

132.8 t/s

Memory bandwidth bound

3-Year TCO Break-Even

N/A

Cloud API cheaper at current rate

Memory Allocation Breakdown 6.0 GB / 24.0 GB

Base Weights: 5.0 GB

KV Cache: 1.0 GB

System RAM Offload: 0.0 GB

VRAM Overhead vs Context Length

3-Year Cumulative Cost ($)

Deployment Launch Parameters

FROM llama3:8b-instruct-q4_K_M
# Hardware target: 24.0 GB VRAM | Context: 4096 tokens
PARAMETER num_ctx 4096
PARAMETER num_gpu 99

Saved local model memory and speed heuristics

Your browser does not support video.

Read the explanation

The saved hardware model defaults to eight billion parameters and four-bit selection, but uses effective four point five bits per weight. Dividing by eight and multiplying overhead one point one two estimates five point zero four gigabytes. Bars compare pre-overhead four point five and estimated total at one hundred pixels per gigabyte. This is a simplified source estimate, not a measured model allocation or exact quantization format. The source cache assumes point two five megabytes per token for eight billion parameters. Four thousand ninety-six tokens and batch one yield one gigabyte; adding weights gives six point zero four. Bars compare weight and total at one hundred pixels per gigabyte. Architecture, KV-head count, cache precision and runtime overhead are not fully modeled. No model backend or inference request is called. With entered bandwidth eight hundred gigabytes per second, source fit status divides bandwidth by weight footprint and batch, estimating about one hundred fifty-eight point seven three tokens per second below its two-fifty cap. Bars compare estimate and cap at two pixels per token per second. This is a bandwidth heuristic, not a benchmark or promised runtime. Full native JSON/reset and original browser dimensions are preserved; actual hardware performance remains unverified.