Interactive explainer for agent builders

Why Traditional RAG Breaks at Scale — and How Agentic Document Search Differs

How does an agent actually search over a large, distributed set of documents? Classic RAG chunks documents, embeds them, and grabs the top-k nearest vectors. That works until facts split across chunks, answers need multiple hops, or knowledge lives in separate indexes. Explore each failure mode below.

The RAG Pipeline, Animated

A document is split into fixed-size chunks, each embedded into vector space. Drag the sliders: at small chunk sizes the key fact splits across two chunks.

Chunk size 512 tokens Overlap 32

Embedding Space Simulator

200 chunks from three source documents, projected to 2D. Pick a query: the query point animlands, top-k neighbors light cyan, and relevant-but-distant chunks pulse coral — semantic near-misses. Toggle distributed corpora to see cross-index blind spots.

k = 8

Run a preset query to retrieve.

Worked Example: One Question, Two Strategies

"What changed in the Q3 refund policy and who approved it?"

Naive RAG

Embed the whole question as one vector
Top-k hits: refund policy chunks only
Approval email lives in a second store — never queried
Answer: policy change described, approver unknown

Agentic Search

Plan: two sub-questions detected
Search policy corpus for Q3 changes
Follow reference to approval thread
Query the email corpus directly
Synthesize: change + approver, with citations

Known Limits of Traditional RAG

Chunk boundary loss

Facts spanning a split are never retrieved intact; overlap only patches small spans.

Top-k ceiling

If the answer needs the 26th-nearest chunk, k=25 silently misses it.

Stale indexes

Embeddings freeze at index time; updated documents drift from their vectors.

Multi-hop questions

One query vector cannot express "find X, then follow its reference to Y".

Distributed silos

Separate indexes per team or system mean single-index retrieval is blind by design.

Check Your Understanding

The search animation is an illustrative local simulation

Read the explanation

The canvas visualizes a document, embedder and vector space. At chunk size five hundred twelve, its count formula draws five chunks; at one hundred twenty-eight it caps at ten. A key-fact split is labeled whenever size is below five hundred twelve. The overlap control is displayed but does not change that condition or the chunk-count formula. This is a teaching animation, not a real document chunker or embedding operation. The scatter plot uses two hundred fixed synthetic coordinates grouped by index modulo three: sixty-seven policy points, sixty-seven email points and sixty-six runbook points. Distributed mode changes positions and limits candidates to policy group zero. Neighbors are selected by squared distance in the canvas coordinate space. No real index is queried. Relevant misses are randomly illustrated, and the displayed miss counter can count a point twice across two tests. The worked comparison highlights authored timeline steps on a timer. It does not generate a plan, fetch a document or prove that an agent found an answer. Coral near misses and the chunk split explain possible retrieval limitations rather than measured semantic relevance or universal RAG failure. Native canvas buffers and controls are preserved in this local draft; public search correctness remains unverified.

Agentic Document Search Mechanics and Vector Retrieval Limits

Why does single-index dense vector retrieval fail on cross-corpus or multi-step questions, and how does agentic search address this?

Standard vector retrieval converts an entire user prompt into a single embedding vector and retrieves the top-k nearest document chunks from an existing index. When an answer requires corroborating distinct pieces of evidence across disconnected corpora—such as cross-referencing a policy clause with its approval record stored in another database—a single nearest-neighbor pass frequently truncates or omits relevant context. Agentic document search introduces iterative planning: an orchestrator breaks the question into intermediate sub-queries, executes lookups against specific targeted indexes, inspects returned evidence, and follows citations before synthesizing a final answer.

The interactive simulator projects 200 synthetic chunks onto a 2D Euclidean canvas rather than calculating true high-dimensional cosine similarity over transformer embeddings. In addition, the chunk boundary split condition (< 512 tokens) and missed-chunk probabilities are illustrative heuristic rules rather than empirically calibrated benchmark results across production tokenizers.

Try a worked example

Clicking the 'Who approved the update?' preset query button triggers the 2D simulator to calculate Euclidean pixel distances from the query vector to chunk points, highlighting the 8 nearest neighbors in cyan. Toggling 'Distributed corpora: ON' filters retrieval candidates to only the Policy corpus, showing how relevant approval evidence located in the Email archive is omitted from the top-k result set.

Single-Vector vs. Decomposed Multi-Hop Retrieval

In standard dense retrieval pipelines, an incoming question is projected into latent space once. If the user's inquiry contains multiple distinct requirements (for example, identifying both an amendment and its authorizing author), the single vector must compromise its semantic representation between both concepts, potentially ranking crucial background documents below the top-k cutoff.

Agentic search addresses multi-hop queries through workflow decomposition: rather than performing a single nearest-neighbor lookup, the agent formulates an initial query, inspects intermediate results for entity names or pointers, and formulates follow-up queries targeting specific document sources before generating the response.

Index Segmentation and Semantic Blind Spots

Enterprise knowledge bases are frequently partitioned across isolated silos (such as technical runbooks, legal policies, and communication archives). A retriever configured to search a single default index cannot discover supporting context stored in peer repositories unless prompted with an orchestration layer capable of routing queries across distributed stores.