The demo that cost $4 to build can cost $40,000 a month at scale. Model your real traffic below — the 3D towers compare what each model tier costs raw (back row) versus with your optimizations applied (front row).
Reuse, don't resend. If your 3,000-token system prompt is identical on every call, caching bills it at roughly a tenth of the input price after the first request. Structure prompts with the stable part first, volatile part last. Typical saving: 30-70% of input spend.
Most requests don't need the flagship. Classify intent with a small model and escalate only hard cases. At the prices above, small is 25x cheaper than flagship — routing 80% of traffic down-tier cuts the blended rate dramatically. Measure quality with an eval set before and after. Typical saving: 40-80%.
Tokens in is a tax on every call. Trim retrieved chunks, cap history to the last N turns, summarize long sessions, and set max_tokens on output. Halving average input halves input cost linearly — no quality eval required for dead weight. Typical saving: 20-50%.
Set budget alerts and hard caps at the provider level on day one; a retry loop with a bug can 100x your spend overnight.
Log cost per request (tokens x price) as a metric next to latency. Cost regressions are deploys too, and they should fail CI like any other regression.
Exponential backoff on 429s, idempotency keys on writes. Naive retries both raise your bill and get you rate-limited harder.