Inside the Transformer: Visualizing Self-Attention Interactive Sandbox

Explore scaled dot-product attention step-by-step from raw tokens to projection vectors and contextual aggregation.

Attention(Q, K, V) = softmax(QKT / √dk) · V
1. Token Flow & Query Selection Click token to set active Query (qi)
2. Scaled Dot-Product Heatmap (Ai,j) Rows = Queries, Cols = Keys
3. Attention Distribution for 'it'
4. Q, K, V Linear Projections (dk = 4) Active: q_it · k_animal
Query Vector (qi)
[0.0, 0.0, 0.0, 0.0]
Key Vector (kj)
[0.0, 0.0, 0.0, 0.0]
Value Vector (vj)
[0.0, 0.0, 0.0, 0.0]
5. Step-by-Step Dot Product & Softmax Math
Computing projection...
6. Contextual Aggregation Vector (zi = ∑ αij vj)
Context Output (zi)
[0.0, 0.0, 0.0, 0.0]

The contextual representation blends token representations proportional to attention scores. Notice how 'it' pulls the physical properties of 'animal' or 'street'.

Enjoy this tool? Build your own with Super
Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.