Interactive visual exploration of the core Self-Attention mechanism: $Attention(Q, K, V) = \text{softmax}(QK^T / \sqrt{d_k})V$
Transformers replace recurrence with Self-Attention. Each word creates a Query vector to search for relevant context, matches it against all other words' Key vectors, and weights their Value vectors.
Click any token to compute its Query $(Q)$ against all Key $(K)$ tokens.
Rows = Queries $(Q)$, Columns = Keys $(K)$. Darker purple denotes higher attention probability.
Each token embedding vector $x_i$ is multiplied by weight matrices $W^Q, W^K, W^V$ to produce Query ($q_i$), Key ($k_i$), and Value ($v_i$).
Dot product $q_i \cdot k_j$ measures compatibility. Divided by $\sqrt{d_k}$ for stable gradients, then normalized across row via Softmax into probabilities summing to 1.0.
The final representation for each token is a linear combination of all Value vectors weighted by their attention score: $z_i = \sum_j A_{i,j} v_j$.
This tool computes a complete normalized attention matrix from explicit teaching rules, not learned query, key or value vectors. Keep the first preset, it query at index seven, and temperature one. Head one starts every score at zero point five, adds one point two for self, and adds three point five when it attends to animal. Head three instead adds a distance bonus: the maximum of zero and three point five minus zero point nine times index distance. Its self score becomes five point two, its neighboring was score three point one, and tired two point two. The two complete ten-key rows show the actual head-menu action. Every probability is labeled and every bar stays at exactly three hundred forty pixels per unit throughout the scene. These assigned patterns illustrate scoring rules, without claiming trained language understanding or a value-vector result. Keep the positional head and it query. The was and tired scores are three point one and two point two, a difference of zero point nine. The genuine temperature slider divides all scores by temperature before stable softmax. At temperature one, the was to tired probability ratio is exponential of zero point nine, about two point four five nine six. At temperature three, that ratio is exponential of zero point three, about one point three four nine nine. Both full ten-key rows sum to one; increasing temperature spreads probability more evenly across the keys. The displayed cases are exact held distributions, not bars growing from zero without a source action. This ratio follows the assigned score difference and actual slider input. It is not a learned prediction. Keep head one and temperature one, then click tired instead of it. For it as query, the assigned rule gives animal raw score four. For tired as query, the separate rule gives animal and it raw score three point three each. Self still receives one point seven. Stable softmax normalizes each complete row independently. The source computes the whole matrix, while the selected token determines which row appears in the distribution bars. There is no matrix-cell click handler in this version. The editable sentence and actual Copy Matrix JSON button preserve the complete matrix, tokens, head and temperature through the real clipboard API. Its suggested three-to-nine-word label is not a programmatic limit: the first preset has ten tokens. These source-specific facts remain explicit; the video does not claim learned vectors or hidden-state mixing.