How Transformers Work: Self-Attention

Interactive Mechanism

Transformers replace sequential loops with Self-Attention: every token computes a Query ($Q$), Key ($K$), and Value ($V$) to weigh contextual relevance with every other token in parallel.

Example Sentence

Attention Head Perspective

Softmax Temperature: 1.0

1. Select Query Token ($Q$) to Inspect Attention ($QK^T / \sqrt{d_k}$)

Click any token badge

Query Word: it Thickness & Opacity = Attention Weight ($W_{ij}$)

Normalized Attention Weights (Softmax)

Attention Math Breakdown

For token it, the model computes compatibility scores against all Keys ($K$), scales by dimension $\sqrt{d}$, and applies Softmax:

$\text{Score}(Q, K_i) = Q \cdot K_i^T / \sqrt{d_k}$

$\alpha_i = \exp(\text{Score}_i / T) / \sum_j \exp(\text{Score}_j / T)$

$\text{Output Context} = \sum \alpha_i V_i$

Resolving reference...

From authored affinity to normalized attention

Your browser does not support video.

Read the explanation

This teaching page starts from authored affinity tables, not trained query and key vectors. For query it, the coreference table gives animal point six five and tired point one eight. At temperature one, the log scores followed by softmax normalize those base weights. In this illustrative two-key subset alone, probabilities are about seventy-eight point three and twenty-one point seven percent. The actual page includes every token, so its full probabilities differ. The source divides log affinity by temperature before exponential softmax. At temperature point five, this is equivalent to squaring the affinities, yielding point four two two five and point zero three two four. The two-key illustration sharpens to about ninety-two point nine and seven point one percent. Bars share eight hundred pixels per probability. This is authored toy arithmetic, not measured model understanding. The syntactic head mixes forty percent of the authored affinity with sixty percent of a distance rule. Its positional term is one divided by distance plus one point two. At distance zero, sixty percent of that term contributes point five. At distance three, it contributes about point one four two nine. Here bars use one thousand pixels per contribution. The context head instead adds a uniform component. These hand-built perspectives are not learned neural attention heads.