T
How Transformers Work
Explore the Self-Attention mechanism: how tokens query context across entire sequences simultaneously.
Step 1: Select Active Query Token ($Q$)
Click any token to compute its attention weights against all Key ($K$) tokens.
Formula: Softmax((Q × Kᵀ) / √dₖ) × V
Attention Flow Visualization
Showing weights projecting from "it" to all keys
Calculated Attention Scores
1. Query, Key, Value Vectors
Every token projection creates three vectors: Q (what I search for), K (what I contain), and V (the actual information payload to pass forward).
2. Scaled Dot-Product
The model multiplies $Q \cdot K^T$ to measure affinity. Scaling by $1/\sqrt{d_k}$ prevents vanishing gradients before applying Softmax to form probability distributions.
3. Parallel Processing
Unlike Recurrent Neural Networks (RNNs) that process step-by-step, Transformers compute all token relationships in parallel across multiple attention heads.