0.2 (Sharp/Argmax)
2.5 (Uniform/Entropy)
Token Sequence & Active Query Vector (Q_i)
Click any token to inspect its attention distributionAttention Weights: Softmax(Q · Kᵀ / √d_k)
Querying: "it" (Token #7)
Total Weight Σ:
100.0%
Projection & Next-Token Logits FFN + Unembed
Top candidate completions after contextual aggregation:
Mathematical Operation
Attention(Q, K, V) = Softmax(Q·Kᵀ / √d_k) · V
• Scaled by 1/√64 = 0.125 to avoid saturated gradients in large dimensions.
• Temperature τ = 1.0 smooths logit peaks before the final value aggregation.
Transformer Layer Lifecycle
1. Embed & Position
Discrete token IDs map to continuous d-dimensional vectors and combine with sinusoidal/RoPE positional signals.
2. QKV Linear Projections
Input matrix X multiplies projection weights W_q, W_k, W_v across h parallel attention heads.
3. Residual + LayerNorm
X + MHA(X) preserves uninterrupted gradient highways across dozens of stacked transformer blocks.
4. MLP / Feed-Forward
Position-wise two-layer expansion (typically 4x d_model) stores factual and linguistic knowledge patterns.