Chunk
Split documents into passages of roughly 200–800 tokens, often with 10–20% overlap so ideas aren't cut mid-sentence. Chunk size is the #1 quality lever: too big dilutes relevance, too small loses context.
Live demo: ask the vector store
Below is a toy knowledge base — 12 chunks from a fictional company handbook, each embedded as a point in space. Pick a question: it gets embedded too (the orange comet), the nearest chunks by cosine similarity light up, and the top-k get stuffed into the prompt. Drag the 3D space to orbit.
Drag to rotate the embedding space · blue dots = chunks · orange = your query
1 · Pick a question
3 · Retrieved chunks (ranked by cosine similarity)
4 · Augmented prompt sent to the LLM
The six steps, precisely
Embed
An embedding model maps each chunk to a vector — commonly 384 to 3,072 dimensions — where semantic neighbors land close together. "PTO policy" and "vacation days" end up near each other despite sharing zero words.
Store
Vectors go into a vector store (pgvector, Pinecone, Weaviate, FAISS…) with an ANN index like HNSW that finds near-neighbors in milliseconds instead of scanning everything.
Retrieve top-k
The user's question is embedded with the same model, and the store returns the k most similar chunks — typically k = 3–10. Many systems then re-rank those candidates with a cross-encoder for precision.
Augment
Retrieved chunks are pasted into the prompt as context, with instructions like "answer only from the context; say 'I don't know' otherwise." This is what grounds the model and slashes hallucination.
Generate
The LLM answers from the supplied evidence — and can cite chunk IDs. Update the documents, and answers update instantly. No retraining, no fine-tuning cycle.
The math: cosine similarity
Similarity between query vector q and chunk vector c is the cosine of the angle between them:
cos(q, c) = (q · c) / (‖q‖ × ‖c‖) → ranges −1 … 1
Worked example in 3 dimensions: q = (0.9, 0.2, 0.1), c₁ = (0.8, 0.3, 0.0), c₂ = (0.1, 0.1, 0.95).
| Pair | Dot product | Norms | Cosine | Meaning |
|---|---|---|---|---|
| q · c₁ | 0.9×0.8 + 0.2×0.3 + 0.1×0 = 0.78 | 0.930 × 0.854 | 0.982 | Nearly same direction — retrieved |
| q · c₂ | 0.9×0.1 + 0.2×0.1 + 0.1×0.95 = 0.205 | 0.930 × 0.960 | 0.230 | Mostly unrelated — skipped |
Real systems do this in hundreds of dimensions, but the geometry is identical — which is why the 3D demo above is an honest miniature.
Why RAG is a leadership problem, not just a tech one
A RAG system is only as good as the documents behind it — and documents are owned by people. Before wiring up a vector store, an honest readiness check:
Knowledge hygiene
Is your source of truth current, deduplicated, and access-controlled? RAG retrieves stale policy pages just as confidently as fresh ones. Garbage in, confidently-cited garbage out.
Ownership
Who owns retrieval quality? Someone must review "the model said X, the doc says Y" reports weekly. Unowned pipelines rot in about a quarter.
People before pilots
The teams that win start with one painful, document-heavy workflow (support macros, policy Q&A, onboarding) and measure deflection rate — not with a moonshot chatbot for everything.
Measurement
Track groundedness (answers traceable to a retrieved chunk), retrieval hit rate, and "I don't know" honesty. If you can't measure it, you're demo-ing, not deploying.