Token Attention Flow (Select Query Token)
Query Token: "it"Attention Weights Vector (Softmax Score): Thickness = Intensity
Full Attention Matrix ($A = \text{softmax}(QK^T/\sqrt{d_k})$)
Query-Key Dot Product Inspector
Target Word (Key)
Softmax Attention Weight
1. Query, Key, Value Projections
Every token embedding is multiplied by learned weight matrices $W_Q, W_K, W_V$. The Query asks what to look for, the Key advertises properties, and the Value holds content to aggregate.
2. Scaled Dot-Product & Softmax
Compatibilities are calculated via $Q \cdot K^T$ and scaled by $\sqrt{d_k}$ to stabilize gradients during backprop. A softmax normalization turns raw logits into valid probability weights summing to 1.0.
3. Multi-Head Parallelism
Multiple attention heads operate concurrently. One head links pronouns to nouns (coreference), another captures syntax, and others track adjacent modifiers—allowing richer representations.