Transformer Attention Head Token Flow Visualizer

Query Token (Destination):

Sentence Preset:

Attention Flow Arcs & Value Aggregation

Query (Q)

Key (K)

Value (V)

Context Out (Z)

5×5 Softmax Attention Matrix HEAD 1

Row $i$ (Query) attends to Columns $j$ (Keys). Each row sums to 1.00.

1. Query · Key Match Dot Product

Score(i, j) = Q_i · K_j

2. Scale & Temperature Scaling

Logit = (Q · K / √d_k) / T

3. Softmax Distribution Weights α

α_ij = exp(z_j) / Σ exp(z_k)

4. Contextual Vector Weighted Sum

Z_i = Σ_j (α_ij · V_j)

Contextualized Output Representation Z for Token "sat":

Linear mixture of token Values weighted by attention head specialization.

Attention Head Specialization Head 1: Syntactic (Next-token & Verbs)

Softmax Temperature (T) Sharpness: <1 = Argmax peak, >1 = Uniform

1.0

Query, Key, and Value Roles

Every token emits a Query ("What info do I need?"), a Key ("What identity do I offer?"), and a Value ("What data do I send if chosen?"). Dot-product matching calculates relevance.

Temperature & Scaling Factor

Dividing by $\sqrt{d_k}$ stabilizes gradient variance across vector dimensions. Temperature scaling $T$ dynamically controls entropy: low temperatures concentrate attention into single tokens.

Multi-Head Specialization

Different attention heads learn distinct linguistic projections. Head 1 captures local sequential structure (adjacent tokens), whereas Head 2 isolates global semantic coreference links.

Exact source temperature action and weighted output

Your browser does not support video.

Read the explanation

Begin with the original source, not an imagined trained model. Its authored four-dimensional embeddings and query, key, and value matrices are toy parameters. The initial dot product for query sat and key cat is two point zero three nine nine. The operation divides that quantity by square root four, which is two. The resulting score is one point zero one nine nine five. Both exact quantities remain visible at three hundred pixels per score; the highlight does not change their widths. The initial case holds temperature one. Now perform a real tool action: move the temperature slider to zero point two, keeping the same query, head, and preset. The source divides its logits by the new temperature and normalizes the exponentials over all five keys. Compare the two exact distributions. Each column sums to one. The sat key gains weight while the other four keys lose weight. Both columns use three hundred eighty pixels per probability. These are two held source results, not partly grown probability bars. Follow the consequence into the actual value computation. The value vectors stay unchanged. For each case, the source multiplies the corresponding attention weight by each value vector, then adds the five contributions. Here we show only the first of four output components. The contiguous colored segments use nine hundred pixels per value unit. Initially the component is about zero point four nine four seven two nine. After lowering temperature, it becomes about zero point four zero eight five one six. The complete context vector is available through the original JSON export. Authored head names do not establish learned linguistic specialization.