The resulting token representation now absorbs information from the tokens it paid attention to.
1. Why Dot-Product Attention?
Before Transformers (Vaswani et al., 2017), recurrent networks (RNNs/LSTMs) processed text word-by-word sequentially, creating an information bottleneck. Dot-product self-attention allows every token to directly query every other token in parallel, measuring semantic alignment through vector geometry regardless of distance in the sentence.
2. Why Divide by √d_k?
As the key dimension $d_k$ grows large, dot products $Q \cdot K^T$ grow proportionally in magnitude. Extremely large values push the Softmax function into regions with near-zero gradients (the vanishing gradient problem). Dividing by $\sqrt{d_k}$ stabilizes the variance to 1.0, preserving healthy training dynamics.
3. The Power of Multi-Head Projections
A single attention head can only attend to one relationship type at a time. Multi-Head Attention projects $Q, K, V$ into multiple smaller subspaces ($h=8$ or $16$ heads), allowing the model to simultaneously track syntax (Head 1), pronoun references (Head 2), and factual associations (Head 3).
4. Residuals & Layer Normalization
Each multi-head attention block is wrapped in a residual connection: $\text{LayerNorm}(x + \text{Sublayer}(x))$. This highway connection ensures original token semantics are never lost and gradients flow uninterrupted through deep networks with dozens of stacked layers.