Why Neural Networks Hallucinate — Mechanistic Manifold Simulator

Embedding Space Projection & Likelihood Manifold

Click / Drag Seed Query to Steer Trajectory

TRUTH SUBMANIFOLD (GREEN) vs HIGH-DIMENSIONAL VOID (WHITE/RED)

Grounded on Manifold

Scenarios:

Mechanistic Hyperparameters

Step rollout: 12 tokens

Manifold Distance ($d_M$)

0.08 σ

Hallucination Risk

4.2%

Attention Grounding

94.0%

Logit Entropy ($H$)

1.12 nats

Sampling Temperature ($T$) 0.20

Increases entropy over token logits. High temperatures eject rollout outside the narrow truth channel.

Attention Context Weight ($\alpha$) 0.90

Pulls representations back toward verified source tokens. Lower values unleash unconditioned priors.

Polysemantic Crosstalk ($\kappa$) 0.10

Simulates shared neuron activations between distant concepts in compressed representations.

Autoregressive Generation Stream Token 12/20

1. The Curse of the Thin Truth Manifold

In a model with $D = 4096$ embedding dimensions, the true training data occupies a tiny, manifold of much lower intrinsic dimension ($k \ll D$). Outside this calibrated corridor lies an exponentially vast vacuum—the Off-Distribution Void.

P(\text{Void}) = 1 - \frac{\text{Vol}(\mathcal{M}_{\text{truth}})}{\text{Vol}(\mathbb{R}^D)} \approx 1 - \epsilon^{D-k} \rightarrow 1.0

Because softmax normalizes logits across all vocabulary tokens, the model is mathematically forced to assign probabilities totaling 1.0 even when the hidden state is adrift in arbitrary, unsupported latent space.

2. Error Snowballing in Autoregression

LLMs sample sequentially: $P(y_1, y_2, \dots, y_N \mid x) = \prod_{t=1}^N P(y_t \mid x, y_{ \tau$.

y_{t} \sim \text{Softmax}\left(\frac{W_U \cdot h_t}{T}\right), \quad h_{t+1} = \text{TransformerLayer}(h_t, \text{KV}_{

Once off the manifold, the attention heads attend to the hallucinated token as if it were ground truth, driving the probability of further fabrications exponentially higher.