How Transformers Work: Interactive Self-Attention & Architecture Lab

Preset Sentence

Attention Head

Token-to-Token Attention Flow Query: "it" (Token #8)

Scaled Dot-Product Attention Heatmap: Softmax(Q Kᵀ / √dₖ) Click any cell to inspect exact Q·K math

Softmax Temperature (τ): 1.00 Lower = sharper focus; Higher = uniform blend

Query & Key Dot Product q₈ · k₁

Query (Q) Vector Token #8 "it"

[+0.42, -0.18, +0.91, -0.05, ...]

Key (K) Vector Token #1 "animal"

[+0.38, -0.15, +0.87, -0.02, ...]

1. Raw Dot Product (q · k): +4.28

2. Dimension scaling (√dₖ = √16 = 4.0): +1.07

3. Temperature applied (τ = 1.00): +1.07

4. Softmax Attention Weight (α): 0.642 (64.2%)

5. Weighted Value contribution (α · v): Norm: 0.81

Why Transformers Dominate Modern AI: Unlike older Recurrent Neural Networks (RNNs) that processed text sequentially one word at a time, Transformers process all tokens in parallel. Multi-head self-attention lets every token directly query every other token in constant $O(1)$ path length, eliminating vanishing gradients over long horizons.