Recurrent Neural Networks (RNNs & LSTMs) processed text sequentially from left to right, creating two severe bottlenecks: information decay over long context windows, and inability to parallelize across modern GPU clusters. Attention computes direct pairwise relationships between every token in O(1) sequential time.
A single attention head can only focus on one type of relationship at a time. By projecting into multiple lower-dimensional subspaces (heads), one head tracks syntactic dependencies (verb-object), another resolves pronouns ("it" → "animal"), and another captures positional adjacency.
As the vector dimensionality $d_k$ grows large, dot products increase in magnitude, pushing the softmax function into regions with near-zero gradients (vanishing gradient problem). Dividing dot products by $\sqrt{d_k}$ keeps variance at 1.0 and stabilizes gradient flow during backpropagation.