Transformer & Self-Attention Explainer

Interactive query-key matching and attention matrix

Attention Is All You Need (2017)

Sentence Sample

Attention Head Focus

Softmax Temperature (τ = 1.0)

SELECT ACTIVE TOKEN (QUERY):

Attention Computation: it

Attention(Q, K, V) = softmax(Q · Kᵀ / √dₖ) V

Softmax distribution across all Key tokens:

1. Parallelization vs RNNs

Traditional RNNs process words sequentially step-by-step, making them slow and vulnerable to vanishing gradients over long distances. Transformers process all tokens simultaneously in parallel.

2. Query, Key, & Value Vectors

Each token is projected into three representations: Query (Q) (what am I looking for?), Key (K) (what do I contain?), and Value (V) (what content do I pass along if matched?).

3. Multi-Head Attention

Multiple attention heads operate in parallel. One head learns pronoun references ("it" ➔ "animal"), another tracks grammar, while others capture topical polysemy ("river bank" vs "financial bank").