Super Super

Transformer Attention Explorer

Self-attention, live

Attention map — rows attend to columns

0 → 1 weight row = query token column = key token

Type any sentence up to 12 tokens. Everything is computed locally in your browser.

How to read this

Each row is a token acting as a query. Each column is a token acting as a key. A darker cell means the query token places more attention weight on that key token. Every row sums to 1 because the scores pass through a softmax.

The four heads use different hand-set projections, so they highlight different relationships: local neighbours, subject–verb links, object links, and positional patterns. Real models learn these projections from data; here they are fixed so you can see the mechanics.

The arithmetic is the real attention formula: softmax(QKᵀ / √d) V. The weights are illustrative, not trained.

What a transformer actually does with your sentence

A transformer is a neural network architecture built around one operation: self-attention. Instead of reading a sentence strictly left to right, self-attention lets every token look at every other token at once and decide how much each one matters. That single idea is what made models like BERT and GPT practical, and it is what this page lets you poke at directly.

The tool above is not a trained model. It runs a small, deterministic, hand-set projection in your browser so you can watch the mechanics of attention without downloading gigabytes of weights. The dot products, the scaling, the softmax and the weighted sums are the real formula. The projections are illustrative.

The three vectors: query, key, value

For each token, the layer computes three vectors by multiplying the token's embedding by three learned matrices:

  • Query (Q) — "what am I looking for?"
  • Key (K) — "what do I offer to others?"
  • Value (V) — "what information do I pass along if I'm attended to?"

The query of one token is compared against the keys of all tokens. A high dot product means the query and key point in a similar direction, so that pair gets a high score.

The attention formula

Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V

Read it in four steps:

  1. Score. Multiply Q by the transpose of K to get a matrix of pairwise dot products.
  2. Scale. Divide by the square root of the key dimension d_k. Without this, dot products of random vectors grow with dimension, the softmax saturates, and gradients vanish.
  3. Normalize. Apply softmax across each row so the scores become weights that sum to one.
  4. Mix. Multiply the weights by V. Each token's output is a weighted blend of the value vectors it attended to.

Why multiple heads

A single attention pattern can only express one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own Q, K and V projections, then concatenates the outputs and projects them back to the model dimension. One head might track local neighbours, another subject–verb agreement, another long-range references. The tool above shows four such heads side by side so you can compare them.

What this tool does and does not show

ShowsDoes not show
Real dot-product scoring, scaling, softmax and weighted sumsTrained weights from a production model
How different projections produce different attention patternsFull multi-layer transformer stacks, layer norm, residual streams or feed-forward blocks
How to read an attention heatmap row by rowAny claim about what a specific commercial model "thinks"

Frequently asked questions

What does self-attention actually compute?

For each token, self-attention builds a query, a key and a value vector. It scores the query against every key with a dot product, scales by the square root of the head dimension, applies a softmax so the scores become weights that sum to one, then returns a weighted sum of the value vectors. The result is a new representation for each token that mixes in information from the tokens it attended to.

Why divide by the square root of the head dimension?

Dot products of random vectors grow with their dimension, so without scaling the softmax inputs would be large and the distribution would collapse onto a single token. Dividing by the square root of the head dimension keeps the variance roughly constant, which keeps gradients usable during training.

What do multiple heads add?

Each head has its own learned query, key and value projections, so it can attend to different relationships in the same sentence. The heads run in parallel and their outputs are concatenated and projected back to the model dimension, letting the layer represent several kinds of dependency at once.

Is this tool running a real trained model?

No. It runs a small, deterministic, hand-set projection in your browser so you can see the mechanics of attention without downloading a model. The arithmetic — dot products, scaling, softmax and weighted sums — is the real attention formula; the weights are illustrative, not trained.

Does my text leave my device?

No. Everything is computed locally in your browser with plain JavaScript. The sentence you type is never uploaded.