Interactive explainer
Why LLMs Double Down on Made-Up Citations
Someone asked: when you call a chatbot out for citing something completely made up, why does it so often defend the fake source? Does it view the citation as its work? Short answer: no. There is no ownership, ego, or belief involved.
Step through a real-feeling failure
This is a scripted, simulated conversation built for teaching. Step through each turn, then tap the annotation to see the mechanism at work.
Watch context bend probability
Every token the model writes gets fed back in as input. Here is the context after the fabricated citation, and how likely each style of reply becomes. Flip the toggle to delete the prior claim from context and watch the odds change.
Percentages are illustrative, not measured — the direction of the effect is the point.
It's not ego
"Doubling down" feels like pride because that is how humans behave when they defend claims. But three simple experiments show no self is involved.
1. Fresh chat, same question
Ask the same question in a brand-new conversation and the model may confidently give a different fake citation — or a real one. A being defending "its work" would remember. The model has no memory of the earlier chat; only the context in front of it.
2. Swap the author
Paste the same conversation but edit the transcript so the fake citation was written by "another assistant." The model defends it just as readily. It is defending consistency of the text, not authorship.
3. Push twice, get a fold
Press harder — "that DOI does not exist" — and the model often flips to an apology, sometimes over-apologizing for things it got right. Ego does not collapse on the second nudge; probability distributions shaped by agreeable training data do.
What actually fixes it
Retrieval grounding
Systems that search real databases and quote retrieved documents put genuine sources into context, so the likely continuation is text consistent with real evidence.
Re-verify in a fresh context
Open a new chat and ask: "Does the paper 'Chen et al. 2019, Nature' on X exist?" Without its own prior claim in context, the model is far more likely to say it cannot verify it.
Ask for uncertainty up front
Prompts like "only cite sources you are confident exist, and say so if unsure" put hedging language into context early, raising the probability of honest continuations.
Myth or mechanism? Six quick questions
Pick an answer for each, then grade yourself.
Frequently asked questions
Does the model know it's lying?
No. "Lying" requires a belief about truth and an intent to hide it. The model computes which token is most probable next given its training and the context. Fluent false text and fluent true text are produced by the same process.
Why does it sound so confident?
Training data is full of confident, assertive prose, and human feedback tuning often rewards direct, helpful-sounding answers over hedged ones. Confidence is a style the model learned, not a signal of accuracy.
If it apologizes when I push, was it just being stubborn?
No — pushback text in context makes apologetic continuations more likely, the same way the fake citation made defensive ones likely. Models can even be pressured into retracting correct answers. Same mechanism, opposite direction.
Will bigger models stop hallucinating?
Bigger models hallucinate less often but not zero. Fabrication is inherent to next-token prediction on incomplete knowledge; grounding in retrieved sources and verification steps are the practical mitigations.