Why RAG Retrieves Right but Still Answers Wrong

The interview question

“Your RAG system retrieves the right documents but still gives wrong answers. Retrieval is correct. Context is being passed. The model has the information. Why is the answer wrong?”The question a senior engineer, fluent in LLMs, vector databases, embeddings, and fine-tuning, could not answer.

It stumps candidates because most RAG debugging intuition lives on the retrieval side: recall, chunking, embedding quality, reranking. But once the right text is in the prompt, a second, quieter set of failures takes over on the generation side. That is what the lab below simulates.

Failure-mode lab

A user asks: “What is the refund window for annual plans?” The correct fact (retrieved perfectly every time) is: annual plans get a 45-day refund window. Pick a failure mode to see how the same correct context still produces a wrong answer, then apply the fix.

Lost in the middle, visualized

Long-context models attend most reliably to the start and end of the prompt. Facts buried mid-context get measurably less effective attention. Drag the slider to move the gold document through a 20-chunk context and watch its effective influence on the answer.

The answer the interviewer wanted

A senior-level answer is a debugging path, not a definition. When retrieval is verified correct, walk generation-side causes in this order:

1. Position auditWhere does the gold chunk land in the assembled prompt? Mid-context placement silently degrades use of the fact.
2. Chunk integrityWas the fact split across chunk boundaries so no single retrieved passage is self-sufficient?
3. Conflict scanDo multiple retrieved chunks disagree (stale docs, old policy versions)? The model averages or picks arbitrarily.
4. Parametric overrideDoes the model prefer its pretraining prior over your document, especially for facts that changed recently?
5. Grounding instructionsDoes the prompt actually require answering only from context, with citation, and permission to say it does not know?
6. Decoding and format pressureTemperature, forced JSON schemas, or tight token limits can truncate or distort a correct grounded answer.