Mechanisms of LLM Hallucination

Neural networks do not "lie" maliciously; they generate tokens sequentially by sampling from high-dimensional probability distributions. When a model crosses an entropy tipping point or encounters ungrounded prompts, the autoregressive mechanism conditions future steps on its own confabulated tokens—causing an irreversible cascade.

1. Select Scenario

2. Sampling Hyperparameters

Temperature (τ) 0.70

High τ flattens logits, boosting low-probability tail tokens.

Prior Bias vs Grounding (α) 0.65

Balance between training data syntax priors and verified factual anchors.

Context Attention Decay (λ) 0.15

Rate at which earlier prompt grounding fades in KV cache.

3. Step Scrubber & Mode

Token Step (0 / 12)

Autoregressive Token Stream & Grounding State

Click any token to inspect its logit distribution

Classification: Grounded Fact Divergence Point Hallucination Autoregressive Cascade

Latent Semantic Manifold (2D Projected Vector Space) Hover to probe

Autoregressive Branching & Attention Cascade Beam probability trajectories

Step 0 Vocabulary Logit Distribution ("...")

Shannon Entropy: 0.00 bits

Candidate Token	Raw Logit	Softmax P(w)	Probability Distribution	Type

Cascade Drift Index

0.84

Distance from factual basin

Attention Confabulation

72%

Weight on past hallucinated tokens

Perplexity Spike

14.2

Uncertainty at branch point

Factual Grounding Score

28%

Residual knowledge retention

Four Core Mathematical Causes of Hallucination

1. Next-Token Objective vs Truth

Language models are trained to optimize maximum likelihood estimation: $P(w_t | w_1, \dots, w_{t-1})$. The objective rewards linguistic plausibility and grammatical fluency, not empirical truth verification. If a false continuation has high stylistic coherence, it receives high logit scores.

2. Sampling in Flat Logit Basins

When a prompt asks an unanswerable question or asserts a false premise, the factual probability distribution becomes diffuse (high Shannon entropy). Non-zero sampling temperature (τ > 0) samples from the tail, crossing over saddle points into confabulated attractor basins.

3. Autoregressive Error Snowball

Generation is strictly Markovian in context: token $t+1$ conditions on token $t$. Once a single hallucinated entity or date is generated, the self-attention mechanism treats that fabricated token as verified context ground truth, compounding hallucinations in subsequent steps.

4. Factual Knowledge Compression

World knowledge is lossily compressed into static feed-forward weight parameters ($W_k, W_v$). When the model attempts to interpolate between rarely correlated facts, latent representations blend distinct concepts into convincing but non-existent synthetic memories.