Next-token probability distribution
Pick a prompt and press âSample a tokenâ.
Pick a prompt and press âSample a tokenâ.
A neural network does not look up facts. It predicts the next token from patterns in its training data. When the pattern is thin, the model still produces a distribution â it just spreads probability across many candidates. Sampling from that distribution produces fluent, confident-sounding text that has no grounding. That is a hallucination.
Each bar is a candidate next token. Bar height is the probability the model assigns after temperature scaling. The teal bar is the token actually sampled. When the teal bar is short, the model is inventing.
No. It is a hand-authored toy distribution that mirrors the shape of a real model's softmax output. The math (softmax with temperature, entropy) is real; the probabilities are illustrative.
Because the model always produces a distribution. There is no âI don't knowâ token unless training taught one. Sampling from a flat distribution yields a random, confident-looking token.
Entropy is the average surprise of a sample. Low entropy means the model is concentrated; high entropy means it is spread out and uncertain.