Architecture Explorer

Transformer Lab: Scaled Dot-Product Self-Attention

Inspect exact matrix operations, Q/K/V vector projections, multi-head routing, and temperature scaling in real-time.

Input Sequence (Tokens)

Educational Presets

Multi-Head Specialization (8 Simulated Attention Heads)

Head 1: Coreference & Long-range Semantic Resolution

Softmax Temp (τ): 1.00 Scale (1/√d_k): Active (d=64)

Attention Matrix Heatmap (Softmax(QK^T / √d_k))

Click Row to Inspect Token Query

Query Vector Focus: it

Attention Output Distribution

Query: [q 0 ...q 63] • Key: [k 0 ...k 63] Formula: A ij = exp(Q i •K j / \sqrtd k / τ) / \sum exp(...)

Positional Encoding Spectrum (Sinusoidal + RoPE Harmonics)

Key Architectural Insight

Complexity: O(N²)

In modern LLMs (e.g., LLaMA, GPT-4), Multi-Head Attention projects the input representation into distinct sub-spaces ($Q, K, V$). Each head captures isolated syntactic or semantic relationships. Scaled dot-product prevents vanishing gradients in the softmax layer when vector dimensionality $d_k$ is large.