Why LLMs need RAG
An LLM's weights are frozen at training time: it can't know your documents, yesterday's news, or private data — and it will confidently hallucinate gaps. RAG fixes this by retrieving relevant text at query time and pasting it into the prompt, so answers are grounded in sources you control, updateable without retraining.
Chunking
Documents are split into chunks before embedding. Too small (<150 tokens) and chunks lose context; too big (>1000) and one vector blurs many topics, hurting search precision. A common recipe: 300–600 tokens with 10–15% overlap, split on semantic boundaries (headings, paragraphs) rather than raw character counts.
Embeddings & vector space
An embedding model maps each chunk to a point in high-dimensional space (e.g. 768 or 1536 dims) where similar meaning = nearby points. Similarity is usually cosine: sim(a,b) = a·b / (|a||b|). "Refund policy" and "money-back terms" land close together even though they share no words — that's semantic search's superpower over keywords.
Vector databases
A vector DB stores millions of embeddings and answers "give me the K nearest vectors to this query" in milliseconds using ANN indexes (HNSW graphs, IVF cells) instead of brute force. They also store metadata (source, date, author) so you can filter — e.g. only search contracts from 2025.
Query rewriting
User questions are often terrible search queries ("it broke again??"). Pipelines rewrite them first: expand pronouns using chat history, split multi-part questions, or generate a hypothetical answer and embed that (HyDE). Better query vectors mean better neighbors mean better answers.
The final prompt
Retrieved chunks are stuffed into a template like: Answer using ONLY the context below. Cite sources. Context: {chunks} Question: {query}. Top-K is a real tradeoff: more chunks raise recall but add noise, cost, and "lost in the middle" effects. Watch the ctx-tokens stat as you drag the slider.