How Transformers Work — Interactive Attention & Architecture Engine

Attention Ray Flow & Token Routing

Click any token to inspect its Query focus and outgoing/incoming Attention weights.

INPUT TOKENS (SELECT TO QUERY) Querying: "it" (Token #7)

Attention Matrix & Latent Vectors

Hover or click cell for raw dot product, scale divisor, and softmax output.

Low (0.0)

High (1.0)

Hover cell to inspect dot product

Forward Pass: Next Token Candidates Temp: 0.70

Full Transformer Block Execution Architecture

Click any stage to reveal how hidden representations transform from raw input tokens to contextual vectors and logits.

STAGE 01

Token & Positional Embedding

Discrete vocabulary tokens convert to dense vectors and add sinusoidal or rotary position encodings.

STAGE 02

Multi-Head Self-Attention

Parallel projection heads compute Q, K, V dot-products, letting tokens dynamically attend to all context words.

STAGE 03

Residual Add & LayerNorm

X + Attention(X) skip connection prevents vanishing gradient, followed by RMSNorm or LayerNorm stabilization.

STAGE 04

Feed-Forward (FFN/MLP)

Two-layer linear network with SwiGLU / GELU activation acts as a key-value associative memory bank.

STAGE 05

Residual Add & LayerNorm

Second skip connection combines the original residual stream with the retrieved factual knowledge from the MLP.

STAGE 06

Unembedding & Softmax

Final layer-normed vector projects onto entire vocabulary via LM head; softmax gives next-token probabilities.

Token & Positional Embedding: Unlike recurrent networks (RNNs) that process word-by-word sequentially, transformers ingest all words at once in parallel. Because attention itself has no built-in notion of order, positional vectors are added to inject word sequence information.

Massive Parallelization

RNNs and LSTMs were bottlenecked by step-by-step sequential time dependencies. Self-attention operations are pure matrix multiplications computable simultaneously across thousands of GPU tensor cores.

Direct O(1) Path Length

In a recurrent model, information from token 1 must pass through 100 intermediate hidden steps to reach token 100. In a transformer, any token connects directly to any other token in a single attention layer.

Specialized Multi-Head Roles

Different attention heads learn distinct linguistic patterns: some specialize in pronoun resolution ("it" → "animal"), others in syntax (subject → verb), punctuation, or factual associative recall.