1.0
Token Attention Map
Click any token to inspect its Query vector
Low (0.0)
Medium (0.5)
High (1.0)
Computation Flow
1. Linear Projections:
Each token embedding $X_i$ is multiplied by weight matrices $W_Q, W_K, W_V$ to yield Query ($Q$), Key ($K$), and Value ($V$).
2. Scaled Dot-Product:
Score = (Q · K) / √d_k
Scores how strongly the selected token seeks information from every other token.
3. Softmax Distribution:
Weights = σ(Scores / T)
4. Contextual Output Vector
The new contextual representation is the weighted sum: Σ (Weight_j × V_j). Words gain context-aware meaning dynamically.
Why Self-Attention?
Unlike RNNs that process text sequentially, self-attention processes all tokens in parallel, capturing long-range dependencies instantly.
Multi-Head Architecture
Multiple attention heads allow the model to simultaneously focus on syntax, grammatical relationships, and deep semantic links.
Feed-Forward & Stacking
Attention layers are interleaved with multi-layer perceptrons (FFN) and residual layer normalization across deep transformer stacks.