Attention matrix â
How transformer self-attention actually works
This tool computes real self-attention on the sentence you type. It is not a diagram of a transformer â it is a small, honest attention block running in your browser. Every token is embedded with a deterministic hash-based vector, projected into query, key, and value spaces by fixed pseudo-random matrices, and then scored with the exact scaled dot-product formula used by transformer models.
The three projections
Each token becomes three vectors. The query asks "what am I looking for?" The key answers "what do I contain?" The value carries "what will I contribute if selected?" Attention is the mechanism that lets a query compare itself against every key and then pull a weighted average of the values.
The scaled dot-product
For a query q and key k of dimension d, the raw score is q·k / âd. The division by the square root of the dimension keeps the dot product from growing with dimension, which would otherwise push softmax into a saturated regime where gradients vanish. The tool shows you the raw dot product and the scaled score side by side.
Softmax turns scores into weights
Each row of scores is passed through softmax: softmax(s)_i = exp(s_i) / Σ exp(s_j). The result is a probability distribution over the keys â every row sums to 1. Those weights are what the heatmap shows. A bright cell means the query token is pulling strongly from that key token.
Value mixing
The output for a query token is Σ w_i · v_i â a weighted sum of the value vectors. That output is what flows into the next layer. In a real transformer, this whole block is repeated across many heads and many layers, and each head learns a different pattern: some heads track adjacent words, some track subjectâverb agreement, some track long-range coreference.
What this tool does and does not show
- Does: real dot products, real softmax, real value mixing, real per-head variation, on your own sentence.
- Does: let you inspect any query row and see exactly which keys it attends to and by how much.
- Does not: use a trained model. The projection matrices are deterministic pseudo-random, so the patterns are illustrative of the mechanism, not of a specific trained network.
- Does not: tokenize with a real BPE tokenizer. It splits on whitespace and punctuation, which is enough to demonstrate the math.
Frequently asked questions
Why does the same sentence give the same result every time?
The projection matrices are seeded deterministically from the head and layer index, so the same input always produces the same attention. That is intentional â it makes the tool reproducible and lets you compare heads fairly.
What does the "head" selector change?
Each head uses a different seeded projection matrix, so it produces a different attention pattern. In a trained transformer, heads specialize; here the specialization is simulated by the seed, which lets you see how different projections produce different attention shapes.
Is my sentence sent anywhere?
No. All computation happens in your browser with plain JavaScript. Nothing is uploaded, logged, or stored on a server.
Why is the attention matrix square?
Because self-attention compares every token against every token, including itself. The diagonal is a token attending to itself, which is usually strong in the first layer.
Can I export the numbers?
Yes â the Export button downloads a JSON file containing the tokens, the raw scores, the softmax weights, and the mixed value vectors for the current sentence, head, and layer.