Inside the Transformer: Attention, QKV & Layer Dynamics

Live interactive sandbox modeling dot-product attention $\text{Softmax}(QK^T/\sqrt{d_k})V$, multi-head specialization, and contextual embeddings.

1. Token Attention Matrix N × N Click row to inspect Query
Low weight (0.00)
High weight (1.00)
2. Contextual Flow & Attention Arcs Token: bank

Active Query token routes information from Key tokens:

3. Next-Token Predictions Softmax(Logits / T)
Core Formula: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)V$  |  Residual Block: $x' = \text{LayerNorm}(x + \text{MHA}(x))$
Hidden dim $d_k=64$ • Multi-Head Count $h=3$