1. Token Sequence & Attention Arcs
0 tokens
Multi-Head Projections
d_model=64 | d_k=16
Self-Attention Matrix A = softmax(QKᵀ / √d_k)
2. QKV Vector Math & Feed-Forward Engine
Score(Q_i, K_j) = (Q_i · K_j) / √16 → Softmax → ∑ α_ij V_j
Query (Q)
[0.82, -0.41, 0.19, 0.55]
[0.82, -0.41, 0.19, 0.55]
Key (K)
[0.77, -0.38, 0.22, 0.49]
[0.77, -0.38, 0.22, 0.49]
Value (V)
[0.11, 0.94, -0.32, 0.60]
[0.11, 0.94, -0.32, 0.60]
Attn Out (Z)
[0.45, 0.68, 0.08, 0.51]
[0.45, 0.68, 0.08, 0.51]
3. Sinusoidal Positional Encoding PE(pos, 2i)
PE(pos, 2i)=sin(pos/10000^(2i/d)), PE(pos, 2i+1)=cos(pos/10000^(2i/d))
4. Transformer Block Architecture
Multi-Head Attention: Allows tokens to simultaneously attend to information from different representation subspaces at different positions. Shape:
[Batch, Seq_Len, D_Model].