How Transformers Work: Interactive Self-Attention & Architecture Explorer

Input Sequence 8 tokens

Classic Presets

Attention Hyperparameters

Softmax Temperature (T) 1.0

Key Dimension ($\sqrt{d_k}$) 8.0

Layer Flow Stepper

Multi-Head Self-Attention Workspace

Select any query token or attention head below to inspect the mathematical dot product routing that allows tokens to contextualize their meaning.

Click a Token to set as Query ($Q_i$):

Attention Weight Matrix ($A_{i,j}$) Hover cell for $Q_i \cdot K_j$ details

Dot-Product Computation LIVE

$A_{i,j} = \mathrm{softmax}\left(\frac{Q_i \cdot K_j^T}{\sqrt{d_k} \cdot T}\right)$

Context Update ($Z_i$): The final output representation for the query token is the weighted sum: $Z_i = \sum_j A_{i,j} V_j$.

Full Transformer Block Forward Pass

Step 1Input

Token + Pos Embed

Converts discrete token IDs into dense vectors $x_i \in \mathbb{R}^{d_{model}}$ + sine/cosine positional signal.

Step 2Projections

Query, Key, Value

Multiplies $X$ by learnable matrices $W_Q, W_K, W_V$ to generate tailored search representations.

Step 3Routing

Scaled Self-Attention

Computes compatibility scores $QK^T/\sqrt{d_k}$, normalizes with softmax, and blends Value vectors.

Step 4Merge

Multi-Head Output

Concatenates attention heads $[head_1; \dots; head_h]$ and projects through output matrix $W_O$.

Step 5Stability

Residual & LayerNorm

Adds skip connection $x + \mathrm{SubLayer}(x)$ followed by mean/variance normalization to stabilize gradients.

Step 6Transform

Feed-Forward MLP

Applies two-layer MLP with expansion (usually $4 \times d_{model}$) and activation ($\mathrm{GELU}$ / $\mathrm{SwiGLU}$).

Key Conceptual Pillars

🔑 Query, Key & Value Intuition

Think of self-attention like an information retrieval database: the Query ($Q$) represents what a token is looking for; the Key ($K$) represents what each token offers or advertises; and the Value ($V$) is the actual substantive semantic content delivered when a match occurs.

🔀 Why Multi-Head Attention?

A single attention head can only focus on one relationship at a time. Multiple heads allow the model to simultaneously track syntactic dependencies (e.g. subject-verb agreement), entity coreferences (pronoun resolution), and positional relationships across different semantic sub-spaces.

⚖️ Why Scale by $\sqrt{d_k}$?

For large vector dimensions $d_k$, the dot product $Q \cdot K^T$ grows large in magnitude. Large inputs push softmax into regions with vanishingly small gradients. Dividing by $\sqrt{d_k}$ preserves unit variance and maintains healthy gradient flow during training.