How Transformers Work: Step-by-Step Attention & Generation

Transformers replace recurrence with Self-Attention: every token projects into Query, Key, and Value vectors. The dot products of Queries and Keys create dynamic weights, mixing context across all tokens in parallel.

Prompt & Head Controls 8 Tokens

Benchmark Sentences

Edit Input Text Press Update

Active Attention Head Semantic / Coreference

Softmax Temperature (τ): 1.0 Sharpness

Autoregressive Generator Predict next token

Interactive Sequence & Attention Matrix Click any token to inspect its Query route

Attention Weights Softmax(Q·Kᵀ / √dₖ) Query: 'it' (pos 7)

Attention(Q₇, K, V) = ∑ Softmax(Q₇ · Kᵢᵀ / √4) · Vᵢ

Query Q:

Key K:

Value V:

Context:

Token "it" computes high cosine similarity with "animal" (+0.84) in Head 1, resolving pronoun reference before feeding to the feed-forward MLP.

End-to-End Layer Stack Flow Forward Pass Pipeline

Step 1

1. Token + Positional Encoding

Tokens are mapped to vectors $E$ and added to sinusoidal/learned position encodings $P$ so order is preserved.

Step 2

2. Multi-Head Self-Attention

Tokens attend to each other across $h$ parallel subspaces, gathering contextual clues across distant positions.

Step 3

3. Add & Layer Normalization

Residual skip connections $x + \text{SubLayer}(x)$ prevent vanishing gradients and stabilize deep representations.

Step 4

4. Feed-Forward MLP

Position-wise 2-layer MLP with expansion (e.g. 4× hidden dim) and GELU/ReLU activation transforms context.

Step 5

5. Output Logits & Softmax

Final layer un-embeds vectors back to vocabulary size $V$, calculating sampling probabilities for token $t+1$.

Predicted Next Token Distribution $P(w_{t+1} \mid w_{1:t})$ Top Candidates

🔍 Why Queries, Keys & Values?

Think of a database lookup. The Query represents what a word is looking for (e.g., "I am a pronoun, what noun do I replace?"). The Key is what each word advertises (e.g., "I am a singular noun 'animal'"). The dot product $Q \cdot K^T$ scores relevance, which scales the Value (the actual information payload).

⚡ Why Parallel Multi-Head Attention?

Unlike RNNs that process tokens one by one ($O(N)$ sequential steps), Transformers compute all token relationships in parallel ($O(1)$ sequential operations with matrix multiplication). Different heads specialize in syntax, pronouns, punctuation, or semantic modifiers simultaneously.

🔄 Autoregressive Decoding

During text generation (like GPT models), the Transformer uses causal masking so token $t$ cannot look at future tokens $t+1$. Once the next token is sampled from the softmax distribution, it is appended to the prompt, and the whole sequence passes through again.