How Transformers Work: Interactive Self-Attention & Architecture Explorer

Input & Parameters Live Matrix

Example Sentences

Edit Token Sequence 8 tokens

Select Attention Head Coreference & Syntax

Softmax Temperature ($\tau$) 1.0

Lower values sharpen attention to peak weights; higher values spread weights evenly.

Key Dimension ($d_k$ scaling) 64

Divisor $\sqrt{d_k}$ prevents dot products from growing excessively large in high dimensions.

Active Query Focus

Click any token in the viewer to inspect how it attends to every other token.

Query Tokens (Source) Click a token to isolate its attention lines

Key / Value Tokens (Target) Showing all mutual relationships

Step 01 Token + Positional Vector lookup + sinusoidal frequencies preserve order

Step 02 Q, K, V Projection Linear transforms create queries, keys, and values

Step 03 Multi-Head Attention Parallel scaled dot products gather context

Step 04 Add & LayerNorm Residual connection prevents vanishing gradients

Step 05 FFN & Logits Dense feedforward layer and vocabulary projection

Step 1: Token & Positional Encoding

x = TokenEmbed(w) + PosEmbed(pos)

Unlike recurrent networks (RNNs) that process words sequentially, Transformers process all words simultaneously in parallel. To retain the order of words, a positional vector (derived from sine and cosine functions of varying frequencies) is added to the word embedding vector.

Tensors & Computation Trace

Batch Size ($B$): 1

Seq Length ($N$): 8 tokens

Hidden Dim ($d_{model}$): 512

Output Tensor: [1, 8, 512]

Next Token Distribution ($P(w_{t+1} \mid w_{1:t})$)

Top Candidates

The final output vector of the last token passes through a linear projection ($d_{model} \to |V|$) and Softmax to yield next-token probabilities.

Computational Complexity & Scaling

Self-attention compares every token to every other token, requiring $O(N^2)$ memory and compute where $N$ is sequence length.

Total Attention Pairs ($N \times N$): 64 pairs

Active Attention Heads ($h$): 4 heads ($d_k=64$)

Total FLOPs Estimate (Layer 1): ~0.52 MFLOPs

01 / Self-Attention

Dynamic Contextual Meaning

Static embeddings like Word2Vec give the word "bank" one vector. In a Transformer, self-attention lets "bank" attend to "river" or "deposit" to dynamically construct distinct meanings depending on context.

02 / Multi-Head Routing

Specialized Subspace Projections

Multiple heads project queries and keys into different vector subspaces. Head 1 might track pronouns ("it" $\to$ "animal"), while Head 2 tracks syntax ("did" $\to$ "cross"), and Head 3 captures adjacent bigrams.

03 / Residuals & LayerNorm

Gradient Superhighways