Neural Mechanistic Interpretability

Why Do Neural Networks Hallucinate?

Autoregressive language models do not store facts: they iteratively sample tokens from next-token logit distributions. Hallucination occurs when attention fades, logit temperature flattens entropy, and a single low-confidence token starts an irreversible confabulation cascade.

Current Token Entropy
1.82 nats
High uncertainty
Prompt Attention Ratio
28.4%
Recency bias dominant
Hallucination Risk
74.2%
Cascade threshold breached
Factual Grounding Score
0.31
Diverged from prompt facts

Autoregressive Generation Stream (Click token to inspect state)

Grounded Drift Hallucination

Attention Head Weights (Layer 24, Head 7)

Token 8 → Past
Columns indicate prior tokens. Darker cells represent high attention weights. Note how attention to the original prompt drops off after token 5 as attention sinks to generated hallucinations.

Softmax Next-Token Candidates

τ = 0.85
Green tokens are factually consistent with context; red tokens introduce invented entities or contradictory facts.
Cascading Confabulation Detected Irreversible Autoregressive Drift at Step 6
At step 6, the model predicted "Treaty of Alençon" (p=0.29) instead of the historical "Treaty of Utrecht" (p=0.44). Once this ungrounded token became part of the model's own prompt window, subsequent attention heads treated this invented treaty as ground truth, generating fabricated signatory years and phantom delegates with high self-reported confidence.

The Three Structural Pillars of LLM Hallucination

Hallucination is not a bug or random glitch; it is an inherent mathematical property of maximum-likelihood sequence estimation without an explicit external truth-evaluator.

01 / STATISTICAL SAMPLING

Softmax Entropy & Tail Sampling

Language models output unbounded logits $z_i$. Applying softmax with temperature produces $P(w_i) = \frac{e^{z_i/\tau}}{\sum e^{z_j/\tau}}$. When $\tau > 0.7$, the probability distribution flattens. Even when a factual token has highest probability, sampling draws from the probability tail where factually erroneous but grammatically plausible tokens live.

02 / ATTENTION MECHANICS

Context Fading & Attention Sinks

As generation proceeds, dot-product attention scores $\text{Softmax}\left(\frac{Q K^T}{\sqrt{d}}\right)V$ diffuse. Models develop "attention sinks" where massive probability mass clusters on recent punctuation or common connective tokens. The semantic constraint of the user's initial prompt decays, allowing generative drift.

03 / CONFABULATION CASCADE

Autoregressive Error Amplification

Transformers predict token $t_n$ conditioned strictly on $[t_1, \dots, t_{n-1}]$. When an incorrect token is sampled at step $k$, it permanently joins the conditioning prompt. The model cannot backtrack; its attention heads now treat its own hallucination as verified premise, making subsequent confabulation mathematically optimal.

Enjoy this tool? Build your own with Super