Inside the Transformer: Visualizing Self-Attention & Deep Architecture

1. Token Flow & Query Selection Click token to set active Query (q_i)

2. Scaled Dot-Product Heatmap (A_i,j) Rows = Queries, Cols = Keys

3. Attention Distribution for 'it'

4. Q, K, V Linear Projections (d_k = 4) Active: q_it · k_animal

Query Vector (q_i)

[0.0, 0.0, 0.0, 0.0]

Key Vector (k_j)

[0.0, 0.0, 0.0, 0.0]

Value Vector (v_j)

[0.0, 0.0, 0.0, 0.0]

5. Step-by-Step Dot Product & Softmax Math

Computing projection...

6. Contextual Aggregation Vector (z_i = ∑ α_ij v_j)

Context Output (z_i)

[0.0, 0.0, 0.0, 0.0]

The contextual representation blends token representations proportional to attention scores. Notice how 'it' pulls the physical properties of 'animal' or 'street'.