1.0
Attention Distribution ($A = \text{softmax}(QK^T / \sqrt{d_k})$)
Click any token to shift attention queryFull Attention Matrix Heatmap
Self-Attention Step-by-Step
1. Query Vector ($Q$):
2. Dot Product Compatibility ($Q \cdot K^T$):
3. Contextual Output ($Z = A \times V$):
1. Query, Key, & Value
Each token embedding is projected into Query ($Q$), Key ($K$), and Value ($V$) vectors via learned linear transformations. The Query asks for relevant attributes, while Keys broadcast token content.
2. Scaled Dot-Product
Similarity is calculated via dot product $Q \cdot K^T$, scaled by $\sqrt{d_k}$ to prevent saturated gradients, and normalized with $\text{Softmax}$ to generate probabilities summing to 1.0 (100%).
3. Parallel Contextual Blend
Unlike sequential RNNs, Transformers process all words simultaneously in parallel. Value vectors ($V$) are weighted by attention weights to create rich, context-aware embeddings.