Why large language models make things up — and how retrieval, tools, citations and calibrated refusal shrink the answer space. Drag the 3D cloud. Flip the layers. Watch the distribution collapse onto the truth.
An LLM is a next-token sampler: it picks each word from a probability distribution learned from text. There is no internal truth database and no fact-checking step — a fluent wrong answer and a fluent right answer are generated by the exact same mechanism.
When training data is sparse or contradictory for a query, the distribution over completions is wide — the cloud you see with all layers off. Sampling from a wide distribution produces confident-sounding fabrications: invented citations, fake API names, wrong arithmetic.
Before answering, an embedding search pulls the top-k relevant passages from a trusted corpus and pastes them into the prompt. The model now conditions on evidence instead of on fuzzy parametric memory — the biggest single spread-reducer in this lab.
Honest limit: retrieval can fetch the wrong passage, the corpus can be stale or wrong, and the model can still ignore or mangle what it retrieved (“ungrounded synthesis”). RAG shrinks hallucination; it does not eliminate it.
Transformers are terrible calculators — digits are just tokens. Routing arithmetic, dates, code execution and live lookups to deterministic tools replaces sampling with computation: the tool result is injected back into context and simply restated.
Honest limit: the model must decide when to call the tool and must copy the result faithfully. Wrong tool arguments or skipped calls reintroduce errors — tool use fixes a class of failures, not all of them.
Requiring “every sentence cites a retrieved span” changes the decoding objective: claims that can't be attached to a source become expensive to emit. It also makes hallucinations auditable — a human can click the citation and check.
Honest limit: models can cite a real source that doesn't actually support the sentence (citation drift). Citations make errors detectable, not impossible — verification is still on the reader or a checker model.
RLHF and targeted fine-tuning teach the model that at low internal confidence the best action is “I don't know” or a clarifying question. This trims the far tails of the answer cloud — the wild guesses — rather than sharpening the center.
Honest limit: over-calibration causes unhelpful refusals of answerable questions; under-calibration lets guesses through. Confidence estimates from a sampler are themselves imperfect.
“Solved” really means engineered around: constrain the model to a specific set of texts (RAG), hand hard sub-problems to tools, force attribution, and reward abstention. Stack all four and factual error rates drop by an order of magnitude on grounded tasks.
But open-ended generation with no corpus and no tools still samples from the wide cloud. Grounding is a system property, not a model property — remove the scaffolding and hallucination returns.