Transformer Architecture Visualized & Demystified

Interactive mathematical mental model of Multi-Head Self-Attention (Vaswani et al., 2017)

1. Token Context Flow & Attention Links Click token to focus
2. Linear Projections for Token "it" Dim $d_k = 64$

Query (Q)

"What am I seeking?"

||Q||: 2.84

Key (K)

"What do I offer?"

||K||: 2.61

Value (V)

"Content delivered"

||V||: 3.10
Self-Attention Operation: Query vector $Q_{it}$ takes the scaled dot-product against all Key vectors $K_j$, normalized by $\sqrt{d_k}=8.0$. Softmax exponentiates and yields a probability distribution summing to 1.0.
3. Attention Weight Matrix $(A = \text{softmax}(QK^T / \sqrt{d_k}))$
4. Output Context Vector Synthesis
$\mathbf{z}_i = \sum_j \alpha_{ij} \mathbf{v}_j$ (Weighted Value Sum) Attending: animal (68%)

The updated representation combines the original token embedding with contextual information drawn proportionally from highly weighted tokens.