How Transformers Work: Self-Attention

Transformers calculate dynamic attention weights across all tokens in parallel using Query, Key, and Value vectors.

1. Input Sequence

2. Attention Head

3. Softmax Temperature 1.0

Active Query Token (Click to inspect relationships):

Attention Weight Matrix (Softmax(Q · K^T / √d_k))

Rows = Query (Q) | Columns = Key (K)

Focused Token Context Distribution

How Values (V) are Aggregated:

The new contextual embedding for token is computed as the sum of all Value vectors weighted by these exact percentages.

1. Query & Key Projections

Every token embedding is multiplied by learned weight matrices W_q and W_k to produce Query and Key vectors representing what it is looking for and what it contains.

2. Scaled Dot-Product

The dot product Q · K measures compatibility. Dividing by √d_k prevents vanishing gradients in large dimensions before passing through Softmax.

3. Multi-Head Synthesis

Different heads track different linguistic relationships (e.g. coreference vs grammar) simultaneously without sequential recurrence bottleneck.