Interview answer, made touchable

Build-a-Support-Agent 🛠️

“How would you design a customer-support AI agent?” The classic answer names six components — Knowledge Base, RAG retriever, LLM, Ticketing, CRM, Monitoring. Here you can switch each one off, run real scenarios, and watch exactly where the flow breaks. Drag the floor to orbit.

drag to orbit · pinch/wheel to zoom

Architecture components — toggle to break things

active disabled failure point query orb

Run a scenario

Config health

Deflection rate—

Predicted CSAT—

What each block does

Knowledge Base — docs, policies, past resolutions. The ground truth the agent is allowed to speak from.
Retriever (RAG) — embeds the question, pulls the top-k relevant chunks, and hands them to the LLM as context. Kill it and the model answers from vibes.
LLM — reads question + retrieved context, drafts the reply, decides intent (answer / act / escalate).
Ticketing — creates and updates tickets (Zendesk, Jira SD) and is the hand-off rail to humans.
CRM — who is this customer? Plan, order history, lifetime value. Turns generic answers into account-aware ones.
Monitoring — logs every turn, tracks confidence, flags bad answers, feeds the improvement loop.

Escalation decision logic

LLM produces an answer and a confidence score, plus a sentiment read.
If confidence < 0.75 or sentiment ≤ “frustrated” → don’t send.
Create a ticket with a conversation summary, tag the intent, route to the right human queue.
Tell the customer honestly: “I’m connecting you with a specialist” beats a wrong answer every time.

Rule of thumb: the agent’s job is to resolve the easy 60–80% and make the hard 20% easier for humans, not to fake omniscience.

Guardrails against hallucination

Grounding: the LLM may only answer using retrieved KB passages — no passage, no answer.
Citations: every claim links back to a source doc so agents (and auditors) can verify.
Refusal rules: pricing promises, legal advice, and account changes above a threshold are hard-blocked to humans.
Eval loop: sample transcripts weekly, score groundedness, and patch the KB where the agent got creative.

Metrics that matter

Deflection rate — % of conversations resolved with zero human touches. Healthy mature setups: 60–80%.
First-response time — agents answer in seconds; humans in minutes/hours. Biggest instant CSAT win.
CSAT — measured post-conversation; watch it per intent, not just overall.
Escalation accuracy — of the chats sent to humans, how many truly needed one? Both false positives and negatives cost money.