How Transformers Work: Interactive Self-Attention Engine

1. Input Sentence

2. Attention Head Focus

3. Softmax Temperature (Scale) 1.0x

Query Token & Attention Weights Distribution

Click any token to set as Query ($Q$)

SEQUENCE TOKENS:

KEY ATTENTION SCORES ($\text{Softmax}(\frac{Q \cdot K_i}{\sqrt{d_k}})$):

Transformer Mechanism Breakdown

1. Query, Key, and Value ($Q, K, V$)

Each token embedding is multiplied by learned weight matrices $W_Q, W_K, W_V$. The Query ($Q$) asks what to look for, the Key ($K$) advertises what the token holds, and the Value ($V$) contains the actual content passed forward.

2. Scaled Dot-Product Attention

Dotting $Q$ with every $K$ computes raw affinity. Dividing by $\sqrt{d_k}$ prevents gradient saturation, and softmax converts scores into probabilities that sum to exactly 1.0 (100%).

3. Weighted Context Vector ($Z$)

The final representation for query "it" is the sum of all Value vectors weighted by its attention probabilities, capturing dynamic context regardless of distance.

Complete Transformer Layer Flow

1. Input + Positional Encoding

2. Multi-Head Attention

3. Add & Norm (Residual)

4. Feed Forward Network

Power normalization sharpens authored toy weights

Your browser does not support video.

Read the explanation

This page uses authored affinity values rather than learned query-key vectors. With two hypothetical affinities point two and point zero five at temperature one, source power normalization gives probabilities point eight and point two. Bars share one thousand pixels per probability. The source teaching text describes general transformer attention, while this toy uses powers rather than exponential softmax. The source raises affinities to power one divided by temperature with a floor of point zero one. At temperature point five, affinities point two and point zero five become point zero four and point zero zero two five. Their ratio increases from four to sixteen. Bars share twelve thousand five hundred pixels per powered score, giving five hundred and thirty-one point two five pixels. Subsequent normalization makes their probabilities sixteen seventeenths and one seventeenth. These authored heuristics are not trained model estimates. A hypothetical dense attention matrix for four and eight tokens has sixteen and sixty-four query-key positions. Bars share six pixels per position. This actual page shows one selected query distribution and explanatory layer-flow content. It does not execute a whole transformer or calculate a value-vector output. Every preset, head, query token, temperature, full clipboard weight payload and reset must remain intact. Preserve complete teaching flow, native resize and earliest footer placement. Independent scene approval remains separate.