How Transformers Work: Interactive Attention & Architecture Sandbox

Step 1

Tokenize & Embed

Text → Vector + Positional Encoding

Step 2

Q, K, V Projections

Linear transformation into subspaces

Step 3

Scaled Dot-Product

Softmax(QK^T / √d_k) × V

Step 4

Multi-Head Concat

Syntactic, coreference & semantic heads

Step 5

FFN & LM Head

Residual + Norm → Next Token Logits

Input Sandbox Presets or Custom

Input Sentence

Classic Linguistic Presets

Active Attention Head

Softmax Temperature (τ): 1.00

0.1 (Sharp/Argmax) 1.0 (Standard) 2.5 (Uniform)

Head 1 (Coreference Specialist): Learns to resolve ambiguous pronouns (e.g. "it") back to their physical entity antecedents ("animal" vs. "street").

Active Sequence Tokens (Click a Token to Focus Query) 10 Tokens

Under the Hood: Scaled Dot-Product Attention Equation d_k = 64 • Scaling Factor √d_k = 8.0

Attention(Q, K, V) = softmax( (Q × K^T) / √d_k ) × V

1. Query Vector (Q) Token: "it"

Represents what the current token is seeking in context.

[ +0.42, -0.89, +0.15, +0.64, ... ]

2. Top Key Vector (K) Token: "animal"

Represents what addresses other tokens advertise.

[ +0.38, -0.85, +0.11, +0.59, ... ]

3. Value Weighted (V) Output Context

Actual content payload mixed into the residual stream.

[ +0.61, -0.14, +0.92, +0.31, ... ]

Autoregressive Next-Token Prediction

After Multi-Head Attention, Feed-Forward layers and LayerNorm project onto vocab logits:

Why Transformers Changed AI (Core Architectural Principles)

1. Parallelism vs. Recurrence

Unlike RNNs or LSTMs that process tokens sequentially ($O(N)$ sequential operations), Transformers process all tokens simultaneously using matrix multiplications ($O(1)$ sequential operations on GPUs), enabling massive context windows.

2. Direct $O(1)$ Path Length

Information between any two tokens across thousands of words flows in a single attention hop. This completely solves vanishing gradients across long distances without lossy hidden state compression.

3. Multi-Head Specialization

By splitting vectors into multiple subspace heads (e.g., 8 to 128 heads), different heads simultaneously track grammar, pronoun coreference, factual associations, and stylistic patterns without interference.

State exported to clipboard and downloaded!

Enjoy this tool? Build your own with Super