Model Tier Optimizer & Cache Routing Workbench

Workload Profile & Rules

Customer Support Bot

Architecture Presets

Monthly Request Volume 500,000 reqs

Shared Prefix Cache Hit Rate 65%

Prompt prefix caching yields up to 75% input token discount on Luna/Sol. Routing Architecture

Luna Escalation Cutoff Score > 42

Prompts scoring above cutoff escalate to Sol or Astra for complex reasoning.

Active Model Catalog Cached / Base $/M

Luna (Fast) $0.04 / $0.15 in • $0.60 out

Sol (Balanced) $0.30 / $1.20 in • $4.80 out

Astra (Flagship) $1.25 / $5.00 in • $15.00 out

Monthly Cost

$1,482

82.4% vs Astra-only

p95 Latency

284ms

-68% vs Flagship

Cache Hit Savings

$642/mo

325M tokens cached

Tier Mix

68 / 24 / 8

% Luna / Sol / Astra

Routing Flow & Latency Waterfall (First 100 Traces)

Luna (~120ms) Sol (~340ms) Astra (~920ms)

Sample Production Traces & Cascades

Click any trace to inspect escalation path

TRACE #001 Handled by Luna

"Can I update my billing email without losing historical receipts?"

Complexity Score

24 / 100

Cache Prefix Status

HIT (1,240 tok)

Execution Latency

142 ms

Incurred Cost

$0.00014

Baseline Astra Cost

$0.00840

Decision: Luna handled fully without cascade. Cost reduction: 98.3%.

Ready. 500,000 monthly transactions modeled across Sol, Luna, and Astra.

How Multi-Tier Caching & Cascades Scale Workloads

Deploying frontier models like Astra for routine tasks introduces prohibitive latency and cloud budget exhaustion. Sol and Luna deliver balanced reasoning and fast edge execution when paired with intelligent prefix routing.

1. Shared Prefix Prompt Caching

System prompts, multi-shot exemplars, OpenAPI schemas, and retrieval contexts are cached in GPU key-value memory. When requests share an initial prefix:

Luna & Sol: 75% price cut on cached input tokens with near-zero prefill time.
Astra: Essential for long-context tool synthesis and complex multi-step reasoning.
Time-to-First-Token (TTFT): Reduces from ~600ms down to ~45ms on repetitive queries.

2. Speculative & Escalation Cascades

Requests are first attempted or scored by Luna. If the task exhibits high ambiguity, edge-case logic, or low model confidence:

Tier 1 (Luna): Resolves 60–80% of repetitive workflows, text parsing, triage, and structured output.
Tier 2 (Sol): Receives escalated agentic trajectories requiring contextual common-sense reasoning.
Tier 3 (Astra): High-stakes code generation, safety audits, and deep mathematical reasoning.