How Self-Attention Works in Transformers

Explore how tokens calculate Query-Key relevance to resolve contextual meaning.

Preset Sentence

Attention Head Focus

Softmax Temperature 1.0

Select Active Query Token ($Q$) Click any token below to compute its attention scores against all Keys ($K$)

Attention Distribution ($Softmax(Q \cdot K^T / \sqrt{d})$) Query: "it"

Full Attention Matrix

1. Query, Key, Value Vectors

Every token embedding is projected via learned linear weights into three vectors: Query (what the token seeks), Key (what the token contains), and Value (the actual content representation).

2. Scaled Dot-Product

Relevance is computed via the dot product $Q \cdot K^T$ divided by $\sqrt{d_k}$ to prevent gradient vanishing, then converted to probabilities using the Softmax function.

3. Contextual Synthesis

The attention weights scale the Value ($V$) vectors across all tokens and sum them together, allowing words like “it” to absorb specific semantic information from “animal” or “street”.