Interactive Research Lab

Transformer Explainer

Deconstructing multi-head self-attention mechanics: query-key compatibility, scaled dot-product normalization, and contextual representations in real time.

Presets:
Select Source Query Token ($q_i$):
Scaled Attention Matrix $A = \text{softmax}(QK^T / \sqrt{d_k})$ Hover cell to inspect
Mathematical Step-by-Step Query: "it"
1. Dot-Product Similarity ($q_i \cdot k_j^T$)
Comparing projection embeddings in head subspace ($d_k = 64$):
q("it") · k("animal") / √64 = 4.28
2. Softmax Probability Distribution
Normalized exponential weighting across all key tokens:
3. Contextual Value Aggregation
Weighted sum producing updated token state $z_i$:
z_i = ∑ α_{ij} v_j → [0.62 animal + 0.18 street + ...]

How Self-Attention Works

Self-attention allows every word in a sequence to dynamically route information from every other word. Instead of static dictionary vectors, words project into three distinct roles: Query ($Q$) ("What am I seeking?"), Key ($K$) ("What information do I hold?"), and Value ($V$) ("What content do I transmit?").

The affinity between token $i$ and token $j$ is calculated via their dot product scaled by $1/\sqrt{d_k}$ to prevent gradient saturation, then normalized with $\text{Softmax}$ so all incoming weights sum to 1.0. Multi-head attention allows the model to simultaneously track syntactic dependencies, coreference, and positional context across parallel subspaces.