Self-Attention Interaction & Routing
Anatomy of the Transformer Architecture
Direct O(1) Token-to-Token Routing
Unlike RNNs or LSTMs that compress entire sentences through a sequential hidden state $h_t$, Transformers allow every single token to directly look at any other token in $O(1)$ path length, regardless of distance. This eliminates gradient decay across long spans.
Parallel Geometric Subspaces
A single attention head can only attend along one linear subspace. Multi-Head Attention splits the $d_{model}$ vector into $h$ heads ($d_k = d_{model}/h$). This lets Head 1 track pronoun coreference ("it" $\to$ "animal"), while Head 2 tracks syntax ("did" $\to$ "cross").
The Residual Stream as Memory Bus
Attention layers don't overwrite representation; they write additive updates to the residual stream: $x_{l+1} = \text{LayerNorm}(x_l + \text{Attention}(x_l))$. The Feed-Forward Network (FFN) then acts as key-value factual associative memory.