How Transformers Work

Interactive Scaled Dot-Product Self-Attention Model

Live Architecture

1. Example Sentence

2. Attention Head Mode

3. Select Focus Query Token

Softmax Temperature (Scale): 1.0

Attention Scores & Q-K-V Decomposition for Token: -

Query (Q) Vector

[0.00, 0.00, 0.00, 0.00]

"What is this token seeking?"

Top Key (K) Match

"Which token has high overlap?"

Output Representation

$\Sigma(\text{Weight} \times \text{Value})$

The 3 Pillars of Transformer Self-Attention

Attention(Q, K, V) = softmax( (Q × K^T) / √d_k ) × V

1. Vector Projection (Q, K, V): Each input token is turned into three distinct learned vectors: a Query (what it searches for), a Key (what it contains for others), and a Value (the content it shares).

2. Dot-Product Scoring: Multiplying $Q$ of the active token by $K^T$ of every other token measures contextual relevance. Scaling by $\sqrt{d_k}$ prevents gradient saturation, and $softmax$ turns raw scores into normalized probabilities summing to 100%.

3. Multi-Head Parallelism: Different attention heads specialize independently—one head captures grammatical structure, while another resolves coreferences (e.g., linking the pronoun "it" back to "animal").

Enjoy this tool? Build your own with Super