Transformer Attention & Architecture Lab Live Model

Dissecting scaled dot-product attention $\text{Softmax}(Q K^T / \sqrt{d_k})$, multi-head projections, and residual token flow.

Classic Presets:
Scaled Dot-Product Matrix
Current Query Token (Click a token to project its queries):
Token Attention Routing Arcs (Query to Keys):
Stage 2: Q·Kᵀ Vector Decomposition
$d_k = 64$ dimensions
$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$ $\sqrt{d_k} = 8.0$
Query Vector (Q) "it" [t=8]
Target Key (K) "animal" [t=1]
Value Vector (V) "animal" [t=1]

Coreference Resolution in Action

Notice how the pronoun "it" places its dominant attention weight on "animal" (score: ~0.84) when the adjective is "tired". If you switch to the "too wide" preset, the key shifts its query alignment to "street" because wide is a physical property of streets, not animals!

The Transformer Architecture: From Token Input to Probability Distribution

Vaswani et al. (2017)
1. Input Layer
Embedding + Positional Coding
Because transformers possess no recurrent recurrence ($RNN$) or convolutions, token order is injected via sinusoidal signals $PE_{(pos, 2i)} = \sin(pos/10000^{2i/d_{model}})$.
2. Attention Mechanism
Multi-Head Scaled Attention
Tokens project into $h$ distinct representation subspaces. Each head independently attends to syntax, pronouns, subject-verb agreement, and distant factual dependencies.
3. Residual Stream
LayerNorm & Skip Add
$x + \text{SelfAttention}(x)$ prevents vanishing gradients across 32–128 stacked decoder blocks, preserving raw token identities through the deep computational stream.
4. Transformation & Output
Position-Wise Feed-Forward
Identical MLPs applied to each position expand vectors to $4 \times d_{model}$ before projecting back, acting as a key-value associative memory that injects factual knowledge.
Enjoy this tool? Build your own with Super