How Multi-Head Self-Attention Drives Transformers
Every token projects into Query ($Q$), Key ($K$), and Value ($V$) dimensional vectors. Scaled dot-products ($Q \cdot K^T / \sqrt{d_k}$) compute dynamic token relevance, which through Softmax normalizes into context-aggregating attention weights before passing through the residual feed-forward stream.
Sequence & Config 5 tokens
Active Attention Head
Head 1 (Syntax/Coreference)
Head 1
Head 2
Head 3
Head 4
Softmax Temperature ($\tau$)
1.00
Higher $\tau$ distributes attention uniformly; lower $\tau$ forces sharp argmax focus.
Causal Autoregressive Masking
Enables GPT-style unidirectional decoder attention (tokens can only attend to past positions).
Tensor Inspector Token #6 ("it")
Token "it" places 68.4% of its attention weight on "animal" in Head 1, successfully resolving the coreference antecedent via query-key vector alignment.
Query Vector ($Q_{6}$)
||Q|| = 2.41
Key Vector ($K_{target}$)
||K|| = 2.38
Value Projection ($V_{target}$)
||V|| = 1.95
Aggregated Context Vector ($Z_i$)
||Z|| = 2.12
Residual stream combination: $x' = \text{LayerNorm}(x + Z W_O)$, allowing uninterrupted gradient propagation through depth.