Transformer Explorable: Interactive Self-Attention Matrix

Input Sentence / Sequence

Linguistic Presets

Active Attention Head

Softmax Scaling (1/√d_k): 1.0x

Causal Autoregressive Mask (GPT-style)

Query (Q) Key (K) Value (V)

Self-Attention Matrix (A = softmax(QK^T / √d_k))

Click any row token to inspect attention

0.00 (Minimal weight)

1.00 (Maximum weight)

Token Vector Inspection

Token: 0

Select a query token to view its real-time dot products with all key vectors.

Positional Encoding Heatmap (d_model = 8)

Sinusoidal wave vectors injected into embeddings: PE(pos, 2i) = sin(pos/10000^(2i/d))

How Transformer Attention Operates

1. Query & Key Projections Each token vector generates a Query ($Q$), Key ($K$), and Value ($V$). The dot product $Q \cdot K^T$ measures conceptual alignment and relevance between pairs of tokens regardless of spatial distance.

2. Scaled Softmax Normalization Dividing by $\sqrt{d_k}$ prevents large magnitude vectors from pushing the Softmax function into regions with near-zero gradients. Softmax normalizes row weights into a valid probability distribution summing to 1.

3. Multi-Head Specialization Parallel attention heads compute distinct subspace representations simultaneously: one head captures pronoun coreference, another tracks syntax, and others retain local or long-range dependencies.