This page runs a real, deterministic scaled dot-product self-attention computation in your browser. Every token in your sentence becomes a query, key and value vector. The heatmap shows, for each query token (row), how much weight it places on each key token (column) after the softmax. Nothing is sent to a server; the numbers you see are computed locally from your input.
For each token we build a query vector q, a key vector k, and a value vector v. The dot product q·k measures how much a query "asks for" what a key "offers". Dividing by âd_k keeps the logits from growing with dimension, and the temperature T lets you sharpen (T < 1) or flatten (T > 1) the distribution before the softmax. The softmax turns logits into a probability distribution over the sequence, and the weighted sum over values is the output for that position.
A single attention head can only express one pattern of relationships. Real transformers run several heads in parallel with different learned projections, so one head might track syntax while another tracks coreference. Here each head uses a different deterministic projection seed, so switching heads shows you genuinely different attention patterns over the same sentence.
the cat sat on the mat because it was warm and inspect it â a well-formed model often routes pronouns back to their antecedent.This is a faithful implementation of the attention mechanism with deterministic pseudo-random projections, not a trained language model. It shows you the shape of attention â the softmax geometry, the scaling, the multi-head structure â but the weights are not the weights of GPT or BERT. A trained model's projections are learned from data; here they are seeded from a hash of each token so results are reproducible and inspectable. The mechanism is identical; the parameters are illustrative.
Each token is hashed with a small FNV-1a function combined with the head index. The resulting 32-bit seed drives a mulberry32 PRNG that fills the query, key and value vectors with values in [-1, 1]. That means the same token always produces the same vectors for a given head, so you can compare sentences and heads reliably.
Temperature divides the logits before the softmax. Lower temperature makes the distribution peakier (the model becomes more confident and attends to fewer tokens); higher temperature flattens it toward uniform attention. It is the same knob used in sampling and in some attention variants.
Yes. After computing, press Download attention matrix (CSV). The file contains the token list, the head index, the temperature, and the full row-normalized attention matrix so you can inspect it in a spreadsheet or notebook.