Transformer Architecture Visual Lab

Attention is All You Need

Sample Preset

Input Text Prompt

Head / Temperature

Computational Circuit Stage

Click any token or matrix cell to inspect

Sequence Tokens (Click to select query token)

Mathematical Inspector

Select elements to reveal real-time vector arithmetic.

Next-Token Probability Distribution P(w|x)

Scaled Dot-Product Attention

Computes alignment weights: Softmax((Q·Kᵀ)/√d_k)·V. The scale factor √d_k prevents dot products from growing excessively large in high dimensions, preventing vanishing softmax gradients.

Multi-Head Subspaces

Multiple attention heads project representations into distinct subspaces, allowing the model to simultaneously attend to syntax, coreference, and positional dependencies.

Computational Complexity

Standard self-attention scales at O(N²) with sequence length N because each token computes pairwise dot products against every other token in the sequence.