How Transformers Work: Scaled Dot-Product Self-Attention

Explore Query (Q), Key (K), and Value (V) token interactions in parallel

Self-Attention Distribution

Formula: softmax(Q · Kᵀ / √dₖ) × V

Click any token below to set the active Query ($Q$):

Attention Weights from "" to all Keys (K)

Attention Heatmap (N × N)

All token pairs
0.00 (Low attention)
1.00 (High attention)
1

1. Projections (Q, K, V)

Each word vector multiplies separate learned weight matrices $W_Q$, $W_K$, $W_V$ to generate Query (what it seeks), Key (what it contains), and Value (what it passes forward).

2

2. Scaled Dot-Product

Taking $Q \cdot K^T$ measures cosine similarity across all pairs. Dividing by $\sqrt{d_k}$ prevents gradient vanishing before applying the Softmax probability normalization.

3

3. Contextual Synthesis

The resulting attention weights scale each token's Value vector $V$. The sum gives a dynamic, context-enriched representation that replaces static word embeddings.

Reading a toy attention heatmap

Read the explanation

This tool assigns hand-built scores before exponential softmax. It does not compute learned query and key vectors or a weighted value output. For query it, the source coreference rule gives animal score four point three and street score one point six. Here one hundred pixels per score gives bars four hundred thirty and one hundred sixty pixels. General dot products need not be cosine similarities unless vectors are normalized. Their score difference is two point seven. Exponentials make the ratio about fourteen point eight-eight. In this two-key illustration alone, normalized probabilities are about ninety-three point seven and six point three percent. Bars share eight hundred pixels per probability. The actual sentence contains more keys and therefore its complete distribution differs. Selecting a matrix cell changes the active query to that cell row. Read the numeric labels, not only the painted bar length. This source gives probability bars a minimum visible width of three percent, and heatmap cells a minimum opacity of point zero five. A hypothetical one percent probability therefore paints a three percent minimum bar. Here one thousand pixels per probability shows ten pixels of actual probability and thirty pixels of the visibility floor. These minimums preserve visibility; they do not change the computed normalized weights.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.