How Transformers Work — Interactive Architecture & Attention Explorer

Sequence Input Inference Input

Example Sequence

Tokens in Context (9) Click to inspect

Multi-Head Attention Heads Head 0: Coreference

Model Dim ($d_{model}$) 768

Head Dim ($d_k$) 64

Layer Depth Layer 4 / 12

Scaling Factor 1/√64 = 0.125

Softmax Temperature: 1.0

Attention Routing & Weights Layer 4 · Head 0

Query Token ($Q_i$) it [idx 7]

Key Token ($K_j$) animal [idx 1]

Scaled Dot-Product 6.42

Softmax Weight ($\alpha_{ij}$) 0.84

Next-Token Logits Unembedding

Probability distribution for next token given the prompt up to the selected position:

Residual Stream Accumulation $x_{l} = x_{l-1} + \text{SubLayer}$

0. Token + Pos Embedding Norm: 1.00

1. Self-Attention Block Δ +0.38

2. MLP Feed-Forward Δ +0.52

3. Current Layer 4 Output Norm: 2.14

The residual stream acts as an additive memory bus where each layer reads and writes information without destroying earlier token representations.

1. Parallel Self-Attention

Unlike Recurrent Neural Networks (RNNs) that process word by word sequentially, Transformers ingest the entire sequence simultaneously. Every token computes direct pairwise relationships with all other tokens via Query-Key-Value interactions.

QK^T computes similarity; V routes information.

2. Positional Encodings

Because self-attention is permutation-invariant (order-blind), deterministic sinusoidal frequencies or learned positional embeddings are added directly to the input vectors so the network understands word order.

PE(pos, 2i) = sin(pos / 10000^{2i/d})

3. Residual Highways & MLPs

Skip connections allow gradients to flow cleanly through hundreds of layers without vanishing. Each attention step aggregates cross-token relationships, while the two-layer MLP retrieves factual knowledge stored in network weights.

Output = LayerNorm(x + FFN(x))