Super logoSuper

Transformer Attention Playground

scaled dot-product self-attention
1.0

Attention weights — each row is a query token

high low
Ready. Edit the sentence or press Compute attention.

What you are looking at

This page runs a real, deterministic scaled dot-product self-attention computation in your browser. Every token in your sentence becomes a query, key and value vector. The heatmap shows, for each query token (row), how much weight it places on each key token (column) after the softmax. Nothing is sent to a server; the numbers you see are computed locally from your input.

The math, in order

Attention(Q, K, V) = softmax( Q·Kᵀ / (√d_k · T) ) · V

For each token we build a query vector q, a key vector k, and a value vector v. The dot product q·k measures how much a query "asks for" what a key "offers". Dividing by √d_k keeps the logits from growing with dimension, and the temperature T lets you sharpen (T < 1) or flatten (T > 1) the distribution before the softmax. The softmax turns logits into a probability distribution over the sequence, and the weighted sum over values is the output for that position.

Why multiple heads

A single attention head can only express one pattern of relationships. Real transformers run several heads in parallel with different learned projections, so one head might track syntax while another tracks coreference. Here each head uses a different deterministic projection seed, so switching heads shows you genuinely different attention patterns over the same sentence.

How to read the heatmap

What this is not

This is a faithful implementation of the attention mechanism with deterministic pseudo-random projections, not a trained language model. It shows you the shape of attention — the softmax geometry, the scaling, the multi-head structure — but the weights are not the weights of GPT or BERT. A trained model's projections are learned from data; here they are seeded from a hash of each token so results are reproducible and inspectable. The mechanism is identical; the parameters are illustrative.

How is the projection seeded?

Each token is hashed with a small FNV-1a function combined with the head index. The resulting 32-bit seed drives a mulberry32 PRNG that fills the query, key and value vectors with values in [-1, 1]. That means the same token always produces the same vectors for a given head, so you can compare sentences and heads reliably.

Why does the temperature change the result?

Temperature divides the logits before the softmax. Lower temperature makes the distribution peakier (the model becomes more confident and attends to fewer tokens); higher temperature flattens it toward uniform attention. It is the same knob used in sampling and in some attention variants.

Can I export the numbers?

Yes. After computing, press Download attention matrix (CSV). The file contains the token list, the head index, the temperature, and the full row-normalized attention matrix so you can inspect it in a spreadsheet or notebook.