How Transformers Work: Multi-Head Attention & Layerflow Workbench

Input Sentence

Sequence Prompt 8 tokens

Benchmark Presets

Active Attention Head 4-Head Projection

Head 1 Syntactic Relations

Head 2 Coreference ("it" → noun)

Head 3 Local / Positional

Head 4 Semantic Similarity

Softmax Temperature (τ) 1.0×

Divides logits by $\sqrt{d_k} \cdot \tau$. Low τ sharpens focus on top token; high τ diffuses attention.

Transformer Pipeline Stage

Ready: Click any token to inspect its query vector.

Active Sequence Tokens — Click a source token to view incoming/outgoing attention Query Token: None

Attention Weight:

High (1.0)

Hover or click over any token above to isolate its attention arcs. Arcs represent how strongly each token (Query) queries every other token (Key).

Dot Product & Softmax Computation

Selected Pair: Query: [it] → Key: [animal]

Raw Dot Product $(q_i \cdot k_j)$ : +14.28

Scaled Score $\frac{q \cdot k}{\sqrt{d_k} \cdot \tau}$ : +7.14

Softmax Probability ($\alpha_{ij}$) : 0.784 (78.4%)

Subspace Embedding Projections (Head 1)

Query Vector $q_i \in \mathbb{R}^4$:

Key Vector $k_j \in \mathbb{R}^4$:

Context Output $v_{context} = \sum_j \alpha_{ij} v_j$:

Why the Transformer Revolutionized AI

Before 2017, natural language processing relied on recurrent neural networks (RNNs and LSTMs). RNNs passed a hidden state sequentially from left to right, making long-range memory weak and preventing parallel computing on modern GPUs.

1. The Query, Key, and Value Intuition

Think of attention like searching a database. The Query is what a word is looking for. The Key is what each word offers. Their dot product determines relevance. The Value is the actual informational content retrieved and blended into the word's updated representation.

Attention(Q,K,V) = softmax(QK^T / √d_k) V

2. Why Multi-Head Attention Matters

A single attention head can only attend to one relationship type at a time. By splitting into 8, 16, or 64 heads, distinct heads simultaneously track grammar (subject to verb), coreference (pronouns to antecedents), idioms, and factual associations in parallel.

MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O

3. Resolving Ambiguity: The Winograd Schema

In our default preset: "The animal didn't cross the street because it was too tired" vs. "because it was too wide", syntactic rules fail. Only semantic attention resolves whether "it" refers to the animal or the street.

Self-Attention resolves polysemy & context

How Transformers Work: Inside Multi-Head Attention

Input Token & Pos Embeddings

Q, K, V Projections

Scaled Dot-Product & Softmax

Value Weighted Sum & Concat

Residual & LayerNorm

Feed-Forward MLP

Why the Transformer Revolutionized AI

1. The Query, Key, and Value Intuition

2. Why Multi-Head Attention Matters

3. Resolving Ambiguity: The Winograd Schema