Interactive visual exploration of the core Self-Attention mechanism: $Attention(Q, K, V) = \text{softmax}(QK^T / \sqrt{d_k})V$
Transformers replace recurrence with Self-Attention. Each word creates a Query vector to search for relevant context, matches it against all other words' Key vectors, and weights their Value vectors.
Click any token to compute its Query $(Q)$ against all Key $(K)$ tokens.
Rows = Queries $(Q)$, Columns = Keys $(K)$. Darker purple denotes higher attention probability.
Each token embedding vector $x_i$ is multiplied by weight matrices $W^Q, W^K, W^V$ to produce Query ($q_i$), Key ($k_i$), and Value ($v_i$).
Dot product $q_i \cdot k_j$ measures compatibility. Divided by $\sqrt{d_k}$ for stable gradients, then normalized across row via Softmax into probabilities summing to 1.0.
The final representation for each token is a linear combination of all Value vectors weighted by their attention score: $z_i = \sum_j A_{i,j} v_j$.