Transformer Under the Hood
$Attention(Q,K,V)$
Interactive Scaled Dot-Product & Multi-Head Self-Attention Pipeline
Space
Step •
←
→
Heads •
R
Reset
Coreference: The animal didn't cross the street because it was too tired
Polysemy: The bank of the river vs the bank with money
Syntax: Time flies like an arrow in the sky
Copy LaTeX
Reset (R)
Query (Q)
Key (K)
Attention Weight
1. Attention Matrix & Scaled Logits
d_k=64
Softmax( (Q · K^T) / √d_k )
Select a source token to view live query-key projections and dot-product calculations.
Temp:
1.0
2. Output Value Aggregation & Next Token
Residual & LayerNorm:
$z = \mathrm{LN}(x + \mathrm{MHA}(x)) \to \mathrm{LN}(z + \mathrm{FFN}(z))$