Inspect how multi-head self-attention routes contextual information between sequence tokens.
Connecting curves denote attention weight $A_{ij} = \mathrm{softmax}(Q_i K_j^T / \sqrt{d_k})$. Click any token to shift query focus.
Darker cells indicate higher softmax weights for that token pair.
Each token projects Query ($Q$) and Key ($K$) representations. Dot product affinities quantify how much information token $i$ seeks from token $j$.
Raw scaled scores are normalized across each row via Softmax so every query token distributes exactly 100% (1.0) of its attention budget.
The resulting representation vector $Z_i = \sum_j A_{ij} V_j$ fuses contextual information into the target token for subsequent transformer layers.
This page builds attention scores from authored word and position rules, not learned query and key vectors. For query it in the first prompt, the coreference rule gives animal score four point five and tired score three. Here bars use one hundred pixels per raw score. The displayed flows and top-four details represent normalized weights, not calculated value-vector aggregation. Exponential softmax turns the score difference one point five into a ratio about four point four eight. In this illustrative two-key subset alone, animal gets about eighty-one point eight percent and tired eighteen point two percent. The real page normalizes over every prompt token, so its complete probabilities differ. Clicking a matrix cell changes the active query to that row. The previous-token head instead assigns score four to the previous position and score two to the current position. In a two-key illustration that gives about eighty-eight point one versus eleven point nine percent. Bars share eight hundred pixels per probability. Both top and bottom SVG token glyphs select a query, even though their labels describe queries and keys. These are hand-built teaching heads, not measured neural model behavior.