Transformer Attention Head Visualizer

1. Input Sequence

2. Active Query Token (Qᵢ)

3. Scaling Scale (√dₖ) 2.00

0.5 (Sharp) 2.0 (Standard) 5.0 (Diffuse)

Attention Weights & Connection Flow

Click any token badge below to make it the active Query

Step-by-Step Score Computation

Key (Kⱼ)	Dot Product (Q·K)	Scaled (Score/√dₖ)	Softmax Weight (α)

Synthesized Value Vector Output

The output vector zᵢ = Σ (αᵢⱼ · vⱼ) aggregates semantic value projections from all tokens according to their normalized attention weights.

Attention insight: Notice how coreferent pronouns (like "it") assign significant attention probability to their antecedent noun phrase based on query-key compatibility.

Scaling a toy attention distribution

Your browser does not support video.

Read the explanation

This page uses authored word affinities and a hash heuristic, not learned query and key vectors. For the query it, the source gives animal score four point three and street score one point seven. Bars use one hundred pixels per score. Other tokens participate too. The value panel ranks weights; it does not calculate a weighted value vector. Dividing by scale two gives scores two point one five and point eight five. Their difference is one point three. Exponential softmax, restricted to this illustrative two-key subset, gives about seventy-eight point six and twenty-one point four percent. The real page normalizes over all sentence tokens, so its full probabilities differ. At scale point five, the scores become eight point six and three point four. Their difference grows to five point two, sharpening this two-key illustration to about ninety-nine point five and point five percent. Bars share eight hundred pixels per probability. Use the actual controls to compare the complete distribution. This toy teaches normalization, not trained model comprehension.