Transformers process natural language in parallel by learning which words relate to each other.
Type any sentence or pick a scenario below to compute scaled dot-product self-attention matrices (Q, K, V), switch attention heads, and inspect token contextualization in real time.
Query vector compares alignment with each Key vector. Softmax normalizes row sum to exactly 1.0 (100%).
| Target Token (Key) | Raw Dot Prod | Weight (α) |
|---|
Older RNNs (Recurrent Neural Networks) processed words sequentially: step 1, then step 2, etc.
Transformers compute every word simultaneously across time using Positional Encodings added to token embeddings, unlocking massive GPU training throughput.
Each word embedding is projected via linear learned weight matrices into three vectors:
Query (Q) asks "What am I looking for?", Key (K) responds "What information do I have?", and Value (V) carries the actual content passed forward.
A single attention distribution can only focus on one kind of relationship at a time. By splitting projections into multiple independent heads (e.g. 8 or 96 in GPT-4), the model simultaneously tracks syntax (verbs to objects), pronouns (coreference), and semantic disambiguation.
After attention aggregates contextual values, a Residual Connection (x + Sublayer(x)) preserves earlier information and prevents vanishing gradients.
A two-layer feed-forward network then stores factual memories and refines the contextualized tokens.
As the key dimension d_k grows large, dot products grow proportionally in magnitude. Very large inputs into the Softmax function push it into regions with extremely small gradients (the vanishing gradient problem). Dividing by √dk stabilizes variance to 1.0.
In isolation, "bank" has multiple senses (financial institution vs. river embankment). When self-attention calculates high affinity between "bank" and "river", the contextualized vector for "bank" incorporates the value vector of "river", shifting its representation toward the geological sense before passing to subsequent layers.
In generative models, future tokens are masked out with -∞ before the softmax step. This ensures position i can only attend to positions ≤ i, preventing the model from cheating during autoregressive next-token prediction.
