Interactive Deep Architecture Model

How Transformers Work: Interactive Attention Studio

Explore how transformers process context in parallel. Select tokens, scrub attention heads, calculate query-key dot products ($Q \times K^T / \sqrt{d_k}$), and see why coreferences and syntax resolve simultaneously.

Multi-Head Self-Attention Network Focus: "it" (Token #8)
Click any token to inspect its Query (Q) attention distribution:
Query × Key Interaction d_k = 64
Attention(Q, K, V) = softmax(Q K^T / 8) V

For token "it", calculating alignment score with all tokens. Max weight directed to "animal" (86%).

Position Encoding (RoPE / Sinusoidal) pos = 8
PE(pos, 2i) = sin(pos / 10000^{2i/d})

Injected into token embeddings so the model knows word sequence without recurrence.

Feed-Forward Layer (FFN) ReLU(xW1 + b1)W2
Residual: x + LayerNorm(Attn(x))

Projects the attended context to a higher 4× dimension and non-linearly compresses it back into the token vector.

Query Token: "it" • Dominant Attention: "animal" (0.86) • Head: H1 (Coreference) • Layer: Layer 1
1

Parallel Self-Attention vs Recurrence

Unlike legacy RNNs and LSTMs that processed text word-by-word sequentially, Transformers process every token in the sequence simultaneously. Each word emits a Query, Key, and Value vector, matching context in $\mathcal{O}(1)$ sequential steps.

2

Why Multiple Heads Matter

A single attention head can only attend to one relationship type at once. Multi-Head Attention projects representations into 4, 8, or 32 distinct subspaces. One head tracks pronoun coreference (e.g. it → animal), while another tracks prepositional phrases or syntax.

3

Residuals, LayerNorm & FFN

After attention aggregates context vectors, residual skip connections (x + Sublayer(x)) prevent vanishing gradients across deep stacks (e.g. 32 to 128 layers). Feed-Forward MLPs then synthesize this gathered information into factual knowledge representations.