KV CACHE SAVINGS LABMATH.JS 13.2.0 READY

Skip prefill when the prefix is already warm.

Model the source loop with explicit assumptions: a cache hit restores stored KV state, so reusable prompt tokens do not need another prefill pass.

REQUEST → CHECK CACHE → HIT: LOAD KV → SKIP PREFILL → SERVE

REQUESTS REUSABLE PREFIX TOKENS CACHE HIT RATE % PREFILL $ / MILLION TOKENS PREFILL MS / TOKEN

ASSUMPTIONS READY

WORKLOAD RESULT

8,000,000baseline prefill tokens

5,600,000tokens avoided

$2.80prefill cost avoided

224.0 sprefill latency avoided

At 70% hits, 700 of 1,000 requests reuse the prefix instead of recomputing it.

Limitations: excludes cache warming, retrieval/transfer time, eviction, output decoding, and model-specific price changes.

Assumed cache hits reduce modeled prefill work

Your browser does not support video.

Read the explanation

The saved scenario has one thousand requests with eight thousand reusable prefix tokens each. Multiplication yields eight million baseline prefill tokens. Bars share fifty pixels per million tokens: at the entered seventy percent hit rate five point six million tokens avoid another prefill pass. The residual is two point four million. Cache hits are assumed, not measured, and this page neither calls a model nor establishes reusable prefix compatibility. With five point six million avoided tokens, the entered rate of fifty cents per million gives two dollars eighty of prefill cost avoided. Doubling the hypothetical rate to one dollar per million doubles this component to five dollars sixty. Bars share seventy pixels per dollar. This is a conditional input price, not a verified provider price, bill, discount or complete inference cost; output decoding and model price changes are excluded. The source multiplies five point six million avoided tokens by the entered zero point zero four milliseconds per token. Dividing by one thousand yields two hundred twenty four seconds. At fifty percent hits the same assumptions yield one hundred sixty seconds. Bars share one point six pixels per second. These are accumulated modeled prefill work, not observed wall clock speedup, parallel throughput or user latency. Warming, cache transfer, eviction and output decoding are omitted. Invalid inputs retain the earlier result, so export after an invalid run can carry that stale calculation.