Model Tier Optimizer & Cache Routing Workbench

Workload Profile & Rules

Customer Support Bot

Architecture Presets

Monthly Request Volume 500,000 reqs

Shared Prefix Cache Hit Rate 65%

Prompt prefix caching yields up to 75% input token discount on Luna/Sol. Routing Architecture

Luna Escalation Cutoff Score > 42

Prompts scoring above cutoff escalate to Sol or Astra for complex reasoning.

Active Model Catalog Cached / Base $/M

Luna (Fast) $0.04 / $0.15 in • $0.60 out

Sol (Balanced) $0.30 / $1.20 in • $4.80 out

Astra (Flagship) $1.25 / $5.00 in • $15.00 out

Monthly Cost

$1,482

82.4% vs Astra-only

p95 Latency

284ms

-68% vs Flagship

Cache Hit Savings

$642/mo

325M tokens cached

Tier Mix

68 / 24 / 8

% Luna / Sol / Astra

Routing Flow & Latency Waterfall (First 100 Traces)

Luna (~120ms) Sol (~340ms) Astra (~920ms)

Sample Production Traces & Cascades

Click any trace to inspect escalation path

TRACE #001 Handled by Luna

"Can I update my billing email without losing historical receipts?"

Complexity Score

24 / 100

Cache Prefix Status

HIT (1,240 tok)

Execution Latency

142 ms

Incurred Cost

$0.00014

Baseline Astra Cost

$0.00840

Decision: Luna handled fully without cascade. Cost reduction: 98.3%.

Ready. 500,000 monthly transactions modeled across Sol, Luna, and Astra.

How Multi-Tier Caching & Cascades Scale Workloads

Deploying frontier models like Astra for routine tasks introduces prohibitive latency and cloud budget exhaustion. Sol and Luna deliver balanced reasoning and fast edge execution when paired with intelligent prefix routing.

1. Shared Prefix Prompt Caching

System prompts, multi-shot exemplars, OpenAPI schemas, and retrieval contexts are cached in GPU key-value memory. When requests share an initial prefix:

Luna & Sol: 75% price cut on cached input tokens with near-zero prefill time.
Astra: Essential for long-context tool synthesis and complex multi-step reasoning.
Time-to-First-Token (TTFT): Reduces from ~600ms down to ~45ms on repetitive queries.

2. Speculative & Escalation Cascades

Requests are first attempted or scored by Luna. If the task exhibits high ambiguity, edge-case logic, or low model confidence:

Tier 1 (Luna): Resolves 60–80% of repetitive workflows, text parsing, triage, and structured output.
Tier 2 (Sol): Receives escalated agentic trajectories requiring contextual common-sense reasoning.
Tier 3 (Astra): High-stakes code generation, safety audits, and deep mathematical reasoning.

Read source routing thresholds and synthetic trace assumptions

Your browser does not support video.

Read the explanation

This local router assigns tiers from authored request complexity scores. With cutoff forty-two, its cascade sends scores up to sixty-seven to Sol after Luna; its classifier instead uses cutoff plus thirty, reaching seventy-two. Both bars depict those threshold scores at seven pixels per score unit. An illustrative score seventy therefore reaches Astra in cascade but Sol in classifier. These are the written branch rules, not actual model competence or external inference. A sampled cache hit applies an assigned seventy-five percent prefix fraction to that fixture request input length, rounded to whole tokens. An illustrative twelve hundred input tokens split into nine hundred cached and three hundred fresh. Bars show token counts at zero point five pixels per token. Eligibility and hit occurrence are synthetic source data, and prices are static model constants. The page does not call models or prove actual provider cache hits. The source simulates one hundred representative traces and multiplies summed cost by monthly volume divided by one hundred. Five hundred thousand assigned calls use scale factor five thousand; one million use ten thousand. Bars show those multipliers at zero point zero five pixels per factor unit. Its p ninety-five label sorts the authored latency values and uses zero-based index ninety-five, the ninety-sixth observation. These extrapolated costs and latencies remain assumptions, not operational measurements or current prices.