Unlike recurrent networks (RNNs) that process words one-by-one, transformers ingest entire sequences in parallel. Through Query-Key dot products and softmax normalizations, every token computes a custom contextual representation by attending directly to every other token.
How much information this token extracts from each key token in the sequence for the current head:
Tokens have no inherent order. Sinusoidal or learned vectors are added so the model knows position.
Learned projection weights map the static input into three distinct functional roles.
Measures compatibility between what token Q seeks and what token K offers.
The output vector is a weighted mixture of all Value vectors according to attention weights.
At sequence entry, the token "it" only possesses generic linguistic features of an impersonal pronoun.
After self-attention, information from attended tokens ("animal") has been absorbed into the residual stream!
Recurrent networks (RNNs/LSTMs) step sequentially through time steps t = 1, 2, ..., T, which prevents hardware parallelization and leads to vanishing gradients across long horizons. Transformers process all tokens simultaneously using matrix multiplications on GPUs/TPUs.
Why divide by √dk? For large dimensions, dot products grow large in magnitude, pushing the softmax function into regions with tiny gradients (vanishing gradient problem). Dividing by √dk preserves unit variance and smooth optimization.
A single attention head can only focus on one type of relationship at a time. Multi-head attention projects the sequence into h distinct subspaces (e.g. 8, 16, or 64 heads), allowing the network to simultaneously track syntactic dependencies, pronoun coreference, and lexical nuances.
In the classic Winograd schema: "The animal didn't cross the street because it was too tired", the pronoun "it" refers to "animal". If you change the sentence to "because it was too wide", "it" refers to "street". Through training, the Query projection for "it" combined with the adjective ("tired" or "wide") produces a key matching the semantic attributes of animate creatures or spatial barriers respectively.
Modern transformers (GPT-4, Claude, LLaMA) stack between 32 and 128 transformer layers.
The formula Output = LayerNorm(x + SubLayer(x)) ensures the residual stream acts as a high-bandwidth information highway.
Attention heads write updates directly into this stream without overwriting earlier representations, enabling gradient backpropagation through hundreds of layers without degradation.