Transformer Lab v2.4 Interactive

1. Tokens & Positional Wave 8 tokens

Text splits into tokens; static embeddings add sinusoidal positional signals $PE_{(pos, 2i)} = \sin(pos/10000^{2i/d})$.

Positional Frequency Wave Addition pos: 0 | d: 16
x₀ = Embedding(Token) + PE(pos)
2. Multi-Head Self-Attention

Select any source token to inspect $Q \cdot K^T / \sqrt{d_k}$ Softmax weights to all target tokens.

Attention Matrix (Softmax)
Selected Token Q·K Vectors
Q:
K:
V:
Head = Softmax(Q·Kᵀ / √dₖ) · V
3. Add & Norm + MLP Expansion: 4× d_model
1. Residual Addition x' = x + MultiHead(x)
2. Layer Normalization μ=0.00, σ=1.00
3. FFN with GELU Activation GELU(xW₁ + b₁)W₂ + b₂
Hidden Activation Density (d_ff = 64) 88% active
4. Logits & Next-Token Softmax