Transformers calculate dynamic attention weights across all tokens in parallel using Query, Key, and Value vectors.
The new contextual embedding for token is computed as the sum of all Value vectors weighted by these exact percentages.
Every token embedding is multiplied by learned weight matrices W_q and W_k to produce Query and Key vectors representing what it is looking for and what it contains.
The dot product Q · K measures compatibility. Dividing by √d_k prevents vanishing gradients in large dimensions before passing through Softmax.
Different heads track different linguistic relationships (e.g. coreference vs grammar) simultaneously without sequential recurrence bottleneck.
This explorer keeps only the first nine whitespace tokens from a selected sentence. Its coreference, syntactic, and positional scores are authored rules, not learned query and key vectors. The source positional head assigns a token itself score three point five and an adjacent token score two point eight. Bars use one hundred pixels per raw score. The matrix displays row-normalized weights; it does not compute weighted value vectors. At temperature one, the score difference is point seven. In this illustrative two-key subset, exponential softmax gives about sixty-six point eight and thirty-three point two percent. All real sentence tokens participate in the actual page, so its full probabilities differ. Selecting any matrix cell switches the active query to that cell row. At temperature point two, dividing the same score difference by temperature increases it to three point five. The two-key illustration sharpens to about ninety-seven point one and two point nine percent. Bars share eight hundred pixels per probability. Adjust the actual controls to compare complete matrix rows. This is toy numerical behavior, not evidence of trained language understanding.