How Transformers Work: Interactive Attention & Architecture Lab

1. Scaled Dot-Product Attention Heatmap

Head 1: Syntactic Dependencies

Head 1 (Syntactic Dependencies): Specializes in linking prepositions and determiners with their head nouns (e.g. "of" → "bank", "river" → "bank").

2. Vectors: Q, K, V & Context Aggregation

$$\text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

Query Token: "bank" dim $d_k = 4$

Query Vector $Q$ ||Q||=1.42

[0.45, -0.82, 0.91, 0.12]

Key Vector $K$ (Active) token: "river"

[0.51, -0.74, 0.88, 0.05]

Raw Dot Product $Q \cdot K^T$: 2.14

Scaled ($\div \sqrt{4}$): 1.07

Softmax Weight: 48.2%

Attention Distribution from "bank" → Keys:

3. The Complete Transformer Layer Pipeline

A single Transformer block consists of tokenization, positional embeddings, multi-head self-attention, residual stream additions, layer normalization, and a two-layer feed-forward network. Select any stage to inspect its transformation:

Input Embedding + Positional Encoding

PE(pos, 2i)

Multi-Head Self-Attention

Concat(head_i) W_O

Add & LayerNorm (Residual Stream)

LN(x + SubLayer(x))

Feed-Forward Network (FFN)

GELU(x W_1 + b_1) W_2

Unembedding & Softmax Logits

Vocab Probabilities

Input Embedding + Positional Encoding

$\vec{x} = E(\text{token}) + PE(\text{pos})$

Transformers have no recurrence (unlike RNNs) and process all tokens in parallel. Because permutation invariance would erase word order, fixed sinusoidal waves or learned positional vectors are added to each token's static semantic embedding vector:

$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})$
$PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})$

This ensures that "bank" at position 1 carries a mathematically distinct vector signature from "bank" at position 8, allowing attention heads to compute relative distances.

4. Autoregressive Next-Token Generation

Decoder-only models (like GPT-4 and Llama) use causal masking so tokens can only attend to prior positions. Click "Generate Next Token" to compute probabilities and step forward:

Temperature: 0.7

Top Candidate Logits & Probabilities: