Transformer Under the Hood

Interactive Visual Architecture · Self-Contained Matrix Simulator
Use ← → keys to step layers
Presets:
Stage 1: Embedding & Positional Encodings Tokens: 0
X0 = PE(pos, 2i) + TokenEmbedding(Tokeni)
Architectural Mechanics
Hover over tokens or attention cells to inspect mathematical vectors.
Self-Attention vs Cross-Attention +
Self-attention calculates relations within the same sequence (Q, K, V from input). Cross-attention pairs Queries from the decoder with Keys & Values from the encoder (e.g. translation).
Encoder-Only vs Decoder-Only vs T5 +
BERT uses bidirectional encoder attention for understanding. GPT uses causal masked decoder attention for generation. T5/BART combines both with cross-attention.
Quadratic Complexity O(N²) Bottleneck +
Every token calculates a dot product with every other token. An N-token context requires N×N pairwise operations, driving the need for FlashAttention and linear approximations.