1. Positional Encoding
Because attention is permutation-invariant, positional vectors (sinusoidal or learned rotary RoPE) are added to token embeddings so the model knows word order.
PE(pos, 2i) = \sin(pos / 10000^{2i/d})
How Transformers Work
An interactive laboratory for the Attention mechanism ($Attention(Q,K,V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$). Inspect live token embeddings, query-key dot products, attention heads, and probability distributions.
Input Sequence / Prompt
Self-Attention Matrix: Cell $(i, j)$ represents the probability weight $A_{i,j} = \text{softmax}(\frac{Q_i \cdot K_j}{\sqrt{d_k}})$ that token $i$ assigns to context token $j$.
Query Vector ($Q_{selected}$)
--
Top Attended Key ($K_{argmax}$)
--
Aggregated Context Output ($\sum A_{ij} V_j$)
--
Multi-Head Attention Dissection
Syntactic / Next Token
Head 1 focuses on adjacent syntactic relationships and direct word bindings (e.g. adjectives to nouns, prepositions to objects).
Head Projection Formula:
\text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)
Each head projects inputs into a unique $d_k = 16$ subspace to capture independent relational patterns in parallel.
FFN & Next-Token Logits Output
Top Candidates
After Multi-Head Attention, residual addition, LayerNorm, and 2-layer FFN ($\text{GELU}(xW_1 + b_1)W_2 + b_2$), the unembedding matrix produces next-token probabilities:
Understanding the Transformer Core Pipeline
Introduced in "Attention Is All You Need" (Vaswani et al., 2017), the Transformer eliminates recurrent neural network sequential dependencies by allowing every token to communicate directly with every other token in constant $O(1)$ path length.
2. Query, Key, Value
Query ($Q$): What a token is searching for.
Key ($K$): What a token contains.
Value ($V$): The information content transferred when matched.
Scores = \text{softmax}(QK^T / \sqrt{d_k})
3. Multi-Head Routing
Splitting dimensions into $h$ heads lets the model attend simultaneously to syntax, semantics, long-range pronouns, and factual associations.
\text{MultiHead} = \text{Concat}(head_1..head_h)W^O
4. Residuals & FFN
Skip connections preserve gradients across hundreds of layers. The position-wise Feed-Forward Network then acts as a key-value associative memory bank.
x' = \text{LayerNorm}(x + \text{SubLayer}(x))