Transformer Explorable: Interactive Self-Attention Matrix

Input Sentence / Sequence

Linguistic Presets

Active Attention Head

Softmax Scaling (1/√d_k): 1.0x

Causal Autoregressive Mask (GPT-style)

Query (Q) Key (K) Value (V)

Self-Attention Matrix (A = softmax(QK^T / √d_k))

Click any row token to inspect attention

0.00 (Minimal weight)

1.00 (Maximum weight)

Token Vector Inspection

Token: 0

Select a query token to view its real-time dot products with all key vectors.

Positional Encoding Heatmap (d_model = 8)

Sinusoidal wave vectors injected into embeddings: PE(pos, 2i) = sin(pos/10000^(2i/d))

How Transformer Attention Operates

1. Query & Key Projections Each token vector generates a Query ($Q$), Key ($K$), and Value ($V$). The dot product $Q \cdot K^T$ measures conceptual alignment and relevance between pairs of tokens regardless of spatial distance.

2. Scaled Softmax Normalization Dividing by $\sqrt{d_k}$ prevents large magnitude vectors from pushing the Softmax function into regions with near-zero gradients. Softmax normalizes row weights into a valid probability distribution summing to 1.

3. Multi-Head Specialization Parallel attention heads compute distinct subspace representations simultaneously: one head captures pronoun coreference, another tracks syntax, and others retain local or long-range dependencies.

Separate the attention display from learned attention

Your browser does not support video.

Read the explanation

The source adjacent head assigns score three point five to the same position and four point five to a neighbor. The bars compare those scores at one hundred pixels per point. This page generates scores with handwritten token and distance rules; its token hash embedding is not a learned query-key projection. The interface formula communicates attention structure but does not prove that its rule-derived scores were obtained by trained matrix multiplication. With two permitted keys scored three point five and four point five at scale one, softmax gives the neighbor about zero point seven three one one. At scale one half, the neighbor receives about zero point six two two five. The bars use six hundred pixels per probability. The page multiplies scores by its scale input, so larger scale sharpens preferences rather than acting like a larger conventional temperature. For query row zero with two tokens, causal mode sets every future column to negative infinity before exponentiation. The only permitted first key gets weight one and the second zero. The bars use five hundred pixels per probability. For later rows, permitted weights still sum to one. This is a causal-mask demonstration on rule-based scores, not evidence of trained predictions or correct resolution of language ambiguity.