Multi-Head Scaled Dot-Product Attention (Softmax(QKᵀ / √dₖ))
Hover cells to inspect
Interactive Token Attention Arcs
Query token: "it"
Mechanics & Vector Inspector
1. Tokenization & Pos
2. Query Vector [Q]
Hover a row token
3. Key Vector [K]
Hover a column token
4. Raw Dot Product
—
5. Scaled (÷ √4)
—
6. Softmax Weight
—
Step-by-Step Architecture Stack
Embedding & Positional: $x_i = \text{Embed}(w_i) + PE(\text{pos})$
Linear Projections: $Q = XW_Q, K = XW_K, V = XW_V$
Scaled Dot-Product: $A = \text{softmax}(QK^T / \sqrt{d_k})V$
Multi-Head Concat & FFN: $\text{LayerNorm}(x + A) \to \text{FFN}$
🧠 Test Your Intuition: The Coreference Challenge
In "The animal didn't cross the street because it was too tired", what noun does "it" attend to most strongly?