Mechanistic Interpretability Lab

Why Do Neural Networks Hallucinate?

Language models do not possess an internal fact-checker or reality model. They predict the probability of the next token based on learned statistical patterns. When training data is sparse, prompt guidance is ambiguous, or a single spurious token enters the context window, autoregressive attention cascades into convincing falsehoods.

Autoregressive Context Window Grounded State
Tokens: 0 / 32 Avg Entropy: 0.00 nats
Click any generated token to inspect its logit distribution and attention weights.
Prompt Factual Hallucinated Cascade Drift

Candidate Token Logits (Softmax Top-5)

Token at step 0

The Softmax Bottleneck: When temperature is raised, ungrounded tokens in the long tail obtain non-zero probabilities. Once sampled, they enter the context and become fixed truths for future steps.

Uncertainty Telemetry

Real-time metrics
Shannon Entropy (H)
Dispersion across vocabulary
0.24
Hallucination Risk Score
Parametric prior divergence
12%
Cascade Drift Factor
Self-reinforcing attention
1.0x
Grounding Confidence
Prompt-token alignment
98%

Self-Attention Drift & Context Coupling

How downstream tokens weight fabricated past tokens
Step through generation to populate the attention weight coupling matrix.
The Autoregressive Compounding Effect: As seen in the attention weights, once a hallucinated entity appears, self-attention heads disproportionately attend to it (up to 70%+ of attention mass), treating the fabrication as factual grounding.

The Four Mathematical Drivers of Hallucination

Why transformers generate plausible falsehoods with 100% syntactic confidence.

1

Maximum Likelihood vs Truth

Language models are trained on cross-entropy loss over next-token prediction: min -log P(w_t | w_<t). The model learns statistical co-occurrence and fluency, not formal epistemology. A syntactically flawless lie has identical objective value to a historical truth.

2

Parametric Memory Degradation

World knowledge is compressed into billions of floating-point weights (feedforward network parameters). Rare facts, specific dates, or minor citations reside in low-density manifolds. When queried, the network smoothly interpolates between neighboring vectors, creating chimera facts.

3

Attention Cascade Feedback

Autoregressive decoding is strictly causal. If token t produces a hallucinated name, token t+1 through t+n attend to it as grounded ground-truth in the key/value cache. The network is forced to maintain narrative coherence with its own earlier mistake.

Deeper Mechanistic FAQ

Can scaling model parameters or context windows eliminate hallucinations?

No. While scaling parameters improves factual retrieval density for head-distribution queries, empirical tests show hallucination rates plateau or shift to more subtle, insidious errors. Because transformers lack an external ground-truth oracle and separate reasoning from storage, probability dispersion in long-tail tasks remains non-zero.

How does Temperature directly cause hallucinated tokens to be sampled?

The softmax function computes P(i) = exp(z_i / T) / Σ exp(z_j / T). As temperature T increases toward infinity, the distribution approaches uniform randomness, meaning fabricated names or dates with low raw logits suddenly become competitive for selection.

What is Sycophancy and how does it relate to hallucination?

RLHF (Reinforcement Learning from Human Feedback) frequently incentivizes models to agree with human user premises rather than contradict them. If a user asks "Why did Napoleon use an iPhone in Waterloo?", the attention mechanism latches onto the user's premise as ground truth and generates an elaborate fictional justification.

What are the most effective current mitigations in production?

1. Retrieval-Augmented Generation (RAG): Supplying verified documents in the prompt context.
2. Logit Lens & Activation Patching: Inspecting internal residual streams to detect when factual confidence collapses.
3. Self-Consistency & Semantic Entropy: Sampling multiple trajectories at T>0 and measuring whether the underlying semantic meaning clusters or diverges.

Enjoy this tool? Build your own with Super
```