How Transformers Work: Interactive Self-Attention Explorer

Presets:

Select Active Head (Multi-Head Self-Attention)

Token Direct Manipulation Click any token to set Query focus ($Q$)

Full $N \times N$ Attention Matrix ($\text{Softmax}(Q K^T / \sqrt{d_k})$) Rows = Queries ($Q$), Columns = Keys ($K$)

Live Math Inspector

Softmax Temperature ($\tau$ or $\sqrt{d_k}$) 1.00

Lower = sharper peak attention; Higher = uniform diffuse attention.

$\alpha_{i,j} = \frac{\exp(q_i \cdot k_j / \sqrt{d_k})}{\sum_m \exp(q_i \cdot k_m / \sqrt{d_k})}$

Focus Token: it

Vector Projections ($d_k = 4$)

High-dimensional token representation projected via learned weight matrices $W_Q, W_K, W_V$:

Out

The 5 Stages of Transformer Computation

Tokenization & Positional Encoding

Unlike recurrent networks (RNNs) that process word-by-word sequentially, Transformers process all words concurrently. Positional vectors ($\sin$ & $\cos$ frequencies) are added to token embeddings so the model knows word order.

Query, Key, Value Projections

Each token vector is multiplied by three learned projection matrices: $W_Q$ (what it is looking for), $W_K$ (what it contains to match against), and $W_V$ (the actual information payload to pass forward).

Scaled Dot-Product Attention

The similarity score between Query $i$ and Key $j$ is computed as $q_i \cdot k_j / \sqrt{d_k}$. Softmax turns these raw logits into a rigorous probability distribution that sums to exactly 1.

Multi-Head Specialization

Multiple attention heads operate in parallel subspaces. One head tracks pronouns (e.g. resolving "it" to "animal"), another tracks direct objects, and another tracks subject-verb agreement.

Residuals & Feed-Forward (MLP)

The aggregated values $\sum \alpha_j V_j$ are added back to the input vector through a skip connection (LayerNorm), followed by a 2-layer Feed-Forward neural network that synthesizes high-level factual knowledge.