Inside the Transformer: Visual & Mathematical Architecture Simulator

1. Token Sequence & Attention Head Selection

Click any token above to isolate its Query vector $\mathbf{q}_i$. █ Query (Q): What am I looking for? █ Key (K): What do I contain? █ Value (V): What information do I pass forward?

2. Scaled Dot-Product Attention Heatmap & Weights

$N \times N$ Matrix ($d_k=4$)

Y-axis: Query token ($i$) Cell color intensity = $\alpha_{ij} = \text{Softmax}$ Attention Weight X-axis: Key token ($j$)

3. Live Mathematical Arithmetic Inspector

q[it] × k[animal]

Step 1: Scaled Dot Product

Select a cell to view vector dot product

Step 2: Softmax Normalization across Row

$\alpha_{ij} = \exp(S_{ij}) / \sum_k \exp(S_{ik})$

Step 3: Weighted Context Vector $\mathbf{z}_i = \sum \alpha_{ij} \mathbf{v}_j$

Dim	Query $q_i$	Key $k_j$	Value $v_j$	Out $z_i$