1.0
Sharp (ArgMax)
Diffused (Uniform)
Interactive Attention Heatmap Click any token to set as Query
Attention Formula Step-By-Step
1. Match (Q × K∘)
Query vector dot-product Key vectors to score raw affinity.
2. Softmax Normalization
Exponentiate and divide by sum so weights total 100%.
3. Context Vector (∑ αV)
Weighted average of Value vectors creates contextual embedding.
Live Mathematical Vector State
| Token | Raw Dot (Q·K) | Softmax (α) | Value Mix |
|---|
Why Transformers Outperform RNNs
- Parallelization: Unlike sequential RNNs, every token in a Transformer is processed simultaneously.
- Direct Connections: Attention creates direct pathways between distant words without gradient vanishing.
- Dynamic Context: "it" receives a dynamic vector shaped directly by "animal" rather than static dictionary lookups.