Neural networks do not store facts; they calculate high-dimensional probability distributions. Hallucination is not a "bug"—it is the mathematical consequence of lossy compression, attention dispersion, and autoregressive Markov drift where each sampled error becomes ground truth for subsequent tokens.
1. Autoregressive Error Compounding
Transformers predict one token at a time: $P(w_1, \dots, w_n) = \prod P(w_t \mid w_{<t})$. If token $w_k$ deviates from truth due to sampling noise or superposition, it becomes immutable context. The model cannot backtrack; it is forced to condition on its own mistake, creating a mathematically coherent fiction.
2. Polysemantic Superposition
Because neural networks possess more concepts than physical dimensions, multiple features share non-orthogonal activation vectors (superposition). When activating a niche entity, overlapping neurons partially excite ungrounded associations, causing "blended" hybrid facts.
3. Attention Dispersion & Saliency Fade
As context expands or distractor tokens increase, the Softmax denominator $\sum \exp(Q K^T / \sqrt{d})$ dilutes attention weights across hundreds of tokens. Factual anchor tokens lose saliency to high-frequency semantic distractors.
4. Sycophancy & Prior Dominance
Pre-trained statistical patterns prioritize sentence fluency and user-query alignment over epistemic verification. When asked to justify a false claim, the network optimizes for plausible continuation rather than refusal.