How Transformers Work: Interactive Self-Attention Simulator

1. Self-Attention Matrix Query: "it" (token 7)

Click any token to inspect

Presets:

Attention Head: Head 1: Coreference Resolution (it → animal)

Select Active Query Token:

Softmax Attention Weights from Query Token Σ = 1.000

2. QKV Vector Mathematics d_k = 4 dimensions

Live Dot-Product & Softmax

Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V

For active query token "it", calculating dot products with all keys, scaled by 1/√4 = 0.5.

Query Vector q_i = x_i · W_Q Token: "it"

Top Attended Key Vector k_j = x_j · W_K Key: "animal"

Value Vector v_j = x_j · W_V Contributed Info

Dot Product Calculation (q_i · k_j) / √d_k Score: +5.12

(0.82×0.91 + 0.14×0.12 + 0.95×0.88 + 0.20×0.18) × 0.5 = 0.82

■ Why Attention Replaced RNNs

Recurrent Neural Networks (RNNs & LSTMs) processed text sequentially token-by-token, creating an information bottleneck and preventing parallel training on GPUs. Self-attention connects every token to every other token in O(1) sequential steps, allowing massive web-scale dataset pretraining.

■ The Q, K, V Database Analogy

Think of Query (Q) as a search query you enter into YouTube. Keys (K) are the titles and tags of all videos in the archive. Values (V) are the actual video streams. The dot product (Q · K) measures keyword relevance; softmax turns it into a percentage, and we blend the corresponding videos together.

■ Why Scale by 1/√d_k?

As vector dimension d_k increases, the magnitude of dot products grows large. In extreme magnitudes, the softmax function produces extremely small gradients (vanishing gradient problem), stopping the network from learning. Dividing by √d_k stabilizes variance at 1.0.

Active State: Token "it" (Query index 7) | Head: Head 1 (Coreference) | Max Attention Target: "animal" (p = 0.542)