Inspect how multi-head self-attention routes contextual information between sequence tokens.
Connecting curves denote attention weight $A_{ij} = \mathrm{softmax}(Q_i K_j^T / \sqrt{d_k})$. Click any token to shift query focus.
Darker cells indicate higher softmax weights for that token pair.
Each token projects Query ($Q$) and Key ($K$) representations. Dot product affinities quantify how much information token $i$ seeks from token $j$.
Raw scaled scores are normalized across each row via Softmax so every query token distributes exactly 100% (1.0) of its attention budget.
The resulting representation vector $Z_i = \sum_j A_{ij} V_j$ fuses contextual information into the target token for subsequent transformer layers.