Inside the Transformer
Interactive Visual Deep-Dive into Self-Attention & Matrix Projections
Reset
Share URL
Export SVG
Curated Sentences & Ambiguity Presets
"The animal didn't cross the street because it was too tired" (Coreference)
"The chef cooked the hot soup and tasted it" (Subject-Verb-Object)
"The bank of the rapid river vs money bank" (Polysemy)
"Transformers process all tokens simultaneously using self-attention"
Custom sequence...
Attention Head Focus
H1: Coreference
H2: Syntax
H3: Proximity
H4: Semantic
Pipeline Stage
←
1. Embeddings
→
1. Positional Embed
2. Q • K Projection
3. Softmax Matrix
4. V Context Output
Scaled Dot-Product Attention:
Head 1 (Coreference)
Scale: √d_k = 2.0
Head 1 Specialty:
Resolves long-range pronominal binding (e.g. mapping pronouns like
"it"
back to its antecedent
"animal"
).
Tensor Inspector & Algebraic Breakdown
Select a cell
Current Token Vector Representation:
Select token
Embedding Vector (Identity + Sinusoidal Positional):
Query (Q):
Key (K):
Value Context (V):
Hover over any cell in the Attention Matrix to see the live dot-product calculation: $$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$