SuperSuper /

Neural Network Hallucination Lab

Next-token distribution

Q: What is the capital of Australia? A: Canberra
Grounded token Confabulated token Other candidates
Model output
Canberra
Confidence 0.00 Evidence strong Verdict grounded

Why do neural networks hallucinate?

A neural network does not "know" facts the way a database does. It learns a probability distribution over the next token given the tokens before it. When the evidence in the prompt is strong, that distribution concentrates on the correct continuation. When the evidence is thin, contradictory, or absent, the distribution spreads out — and the model still has to emit some token. That token is often fluent, confident, and wrong. That is a hallucination.

What this lab shows

Each bar is a candidate next token. The height is the model's probability for that token. The green bar is the grounded answer; the red bar is a plausible-sounding confabulation. Move the sliders and watch how the distribution reshapes:

Reading the verdict

The verdict compares the sampled token to the grounded answer. A grounded verdict means the model landed on the supported token. A confabulated verdict means the model produced a fluent but unsupported token — the classic hallucination. The confidence number is the probability the model assigned to whatever it emitted, which is why a hallucination can still look confident.

What this model is and is not

This is a small, transparent softmax model over a fixed candidate set. It is a teaching instrument, not a real language model. The probabilities are computed from the parameters you set, not from a trained network. The point is the shape of the distribution and the relationship between evidence, pressure, and confidence — not any specific number.

How is the probability computed?

Each candidate gets a score: score = evidence × grounded + pressure × fluency − diversity × rank, then a softmax with temperature is applied. The grounded candidate receives the evidence bonus; the confabulation receives the pressure bonus. This is a simplified illustration of the tension between retrieval-grounded evidence and next-token fluency.

Why does a confident model still hallucinate?

Because confidence is a property of the distribution, not of the truth. A model can assign 0.9 probability to a wrong token if the training signal, prompt framing, or sampling path favors fluency over factuality. Hallucination is a calibration failure, not a knowledge failure.

What reduces hallucination in real systems?

Retrieval-augmented generation (grounding the prompt in retrieved evidence), lower temperature for factual tasks, explicit uncertainty instructions, and post-hoc verification against sources. None of these eliminate hallucination; they shift the distribution toward grounded tokens.