Input Embeddings (X)
4 × 4
Query Matrix (Q = X · W_q)
4 × 2
Key Matrix (K = X · W_k)
4 × 2
Value Matrix (V = X · W_v)
4 × 2
Raw Scores (Q · K^T / √d_k)
4 × 4
Softmax Attention (A)
4 × 4
Context Vectors (O = A · V)
4 × 2
Step-by-Step Linear Algebra
Click any cell in the Softmax Attention (A) or Raw Scores matrix to dissect its exact scalar vector dot product math.
Row 1 ("bank") → Col 2 ("river")
Q["bank"] = [0.12, 0.84]
K["river"] = [-0.50, 0.30]
Dot = (0.12 × -0.50) + (0.84 × 0.30) = 0.192
K["river"] = [-0.50, 0.30]
Dot = (0.12 × -0.50) + (0.84 × 0.30) = 0.192
√d_k = √2 ≈ 1.414
Score = 0.192 / 1.414 = 0.136
Score = 0.192 / 1.414 = 0.136
exp(0.136) = 1.146
Sum_row exp = 5.342
Attention Weight = 1.146 / 5.342 = 0.215
Sum_row exp = 5.342
Attention Weight = 1.146 / 5.342 = 0.215
"bank" focuses 21.5% of its attention context on "river".
Dominant Pair Analysis
Dominant Pair: bank → money