Transformers process all tokens simultaneously. Each token generates a Query (Q) to search what it needs, matches against other tokens' Keys (K) to compute attention scores, and computes a weighted sum of Values (V) to update its contextual meaning.
Unlike RNNs that process word-by-word sequentially, Transformers ingest the entire sequence at once. Positional encodings are added so the network knows word order.
Dotting Q and K measures compatibility between words regardless of distance. Softmax transforms these raw dot products into normalized probability percentages.
The final representation blends meaning from all related tokens. Multi-Head Attention repeats this across dozens of subspaces (grammar, coreference, semantics).
This interactive page applies stable softmax to authored token affinities. It does not run a trained transformer or compute query, key and value vectors. Hold its first sentence and it query at source index seven. The actual native temperature slider can move from one to two point five. Animal and tired keep assigned raw scores eight point five and seven point two. The code divides all ten scores by the selected temperature, subtracts the largest scaled score for numerical stability, exponentiates and normalizes. At temperature one, animal receives about zero point seven eight one four zero five and tired zero point two one two nine five eight. At two point five, these become about zero point four eight seven eight two two and zero point two nine zero zero two zero. Their ratio changes from about three point six six nine three to one point six eight two zero three. Every held bar uses three hundred forty pixels per normalized weight unit, including tiny weights with readable individual values. The distribution changes because of a genuine native input; no numeric bars falsely grow from zero. A flatter authored example is not a measured learned attention head or a semantic prediction. Keep it selected as the query and temperature one. Change the actual sentence menu from the tired sentence to the wide sentence. Both source options contain ten tokens; the final token changes and the source chooses a different authored affinity map. In the tired case, animal has score eight point five and street one point five. In the wide case, animal has one point two and street eight point eight, with wide seven point six. Stable softmax therefore produces the two full ten-key distributions shown here on one shared scale. Each token is labeled by its source index, so the capitalized The and the lowercase the remain distinct source keys. The source maps are assigned teaching examples. This native sentence change explains the numerical result, but does not establish a learned coreference decision. The original page does not execute value mixing or compute a final hidden representation. All quantities remain held at the exact computed cases rather than suggesting fractional sentence changes or an unsupported animated intermediate distribution. Hold the tired sentence and temperature one. Use the native query menu or token buttons to change the selected query from it, source index seven, to animal, index one. The code now uses the animal affinity map. Its cross score is six point two, tired five point five and it five point zero. Where that map has no explicit key value, the code uses its actual fallback: self score three, or zero point eight plus zero point four times the sine of the key index plus query index. Those fallback entries participate in the same complete ten-key softmax as the assigned entries. Both held distributions show the exact result with every key and numeric weight labeled. The normalization sum is one in each case; no independent value-vector calculation occurs. This causal query action teaches how the authored lookup and fallback rule affect a selected row. It does not train weights, measure model understanding or prove a transformer output. Original controls, native mouse and touch behavior, and responsive dimensions are preserved.