How Transformers Work: Self-Attention

Interactive token attention mapping & scaled dot-product computation

Input Sentence (Tokens)

Attention Head

Softmax Temperature

1.0

TOKEN ATTENTION FLOW & ARCS Click a token to query

Thicker glow lines indicate stronger attention weight from the selected Query token ($Q$) to Key tokens ($K$).

Formula: Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V √dₖ = 8

ATTENTION MATRIX (ALL HEADS) Row = Query (Q), Col = Key (K)

0.0

1.0

1. Query, Key & Value (Q, K, V)

Each word token is projected into three vectors: Query (what am I looking for?), Key (what info do I hold?), and Value (the actual token content).

2. Scaled Dot-Product Attention

Calculating $Q \cdot K^T / \sqrt{d_k}$ measures token compatibility. Softmax normalizes these scores into a valid probability distribution summing to 1.0.

3. Multi-Head Aggregation

Multiple attention heads run concurrently. One head tracks pronouns (e.g. "it" → "animal"), while others capture grammar, verbs, or prepositional phrases.