ARCH-101

Transformer Multi-Head Attention Lab

1. Input & Tokens

Select or type phrase

2. Attention Head

Specialized heads
1.0x
Attention Equation: $\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$ $d_{\text{model}}=64, d_k=16$
Inspecting Focus Token: "it" → Queries matched with all Key vectors

3. Vector Probe

Projections & Norms

Focus Token: "it" Pos #7

• Query Vector ($q_i \in \mathbb{R}^{16}$)
• Key Vector ($k_i \in \mathbb{R}^{16}$)
• Value Vector ($v_i \in \mathbb{R}^{16}$)

Top Attention Weights Σ = 1.00

Contextualized Output Vector

Weighted sum $z_i = \sum_{j} \alpha_{ij} v_j$ passed to Feed-Forward Network:

◆ 1. Queries, Keys & Values

Like a database lookup: every token projects its embedding into a Query (what it's searching for), a Key (what it offers), and a Value (the semantic payload it transmits).

◆ 2. Scaled Dot-Product

The dot product $q_i \cdot k_j$ measures compatibility. Dividing by $\sqrt{d_k}$ prevents gradients from vanishing during backprop, and Softmax converts raw logits into a strict probability distribution summing to 1.0.

◆ 3. Multi-Head Parallelism

Multiple attention heads run simultaneously, allowing the network to simultaneously track pronouns ("it" → "animal"), syntactic verbs, prepositional phrases, and broad document themes in parallel.

◆ 4. Residuals & MLP Layers

After attention aggregates context, residual connections ($x + \text{SubLayer}(x)$) and LayerNorm stabilize signal propagation, followed by a position-wise Feed-Forward Network that enriches token representations.

Enjoy this tool? Build your own with Super