Transformer Inside Out Interactive Visualizer

Inspect self-attention mechanisms, multi-head routing, QKV projections, and residual transformations step by step.

Sentence Presets

Input Sequence (max 10 tokens)

Attention Head Specialization Coreference

Softmax Temperature (T) 1.0

Input Tokens & Embedding Breakdown

Every token is mapped into an initial embedding $E_i \in \mathbb{R}^{d}$ and added with a sinusoidal positional vector $P_i$.

Vector Composition Formula

X[i] = TokenEmbedding(t_i) + PositionalEncoding(i, d)

Select or hover a token above to view its embedding vector.

Positional Frequency Heatmap ($d=8$)

Linear projections transform input $X$ into Query (Q) (what token seeks), Key (K) (what token contains), and Value (V) (content broadcasted).

Query Vector $Q = X \cdot W_Q$

Key Vector $K = X \cdot W_K$

Value Vector $V = X \cdot W_V$

Bipartite Attention Flow

Attention Weight Line

Query Token (Source)

Key Token (Target)

Attention Weight Matrix $A = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k} \cdot T}\right)$

Hover or click a matrix cell to inspect the dot-product & softmax math.

Contextual output $Z = A \cdot V$ passes through a Residual Connection ($X + Z$), LayerNorm, and a 2-layer Feed-Forward Network (FFN).

Contextual Aggregation ($Z = A \cdot V$) vs Residual ($X + Z$)

Representation Shift

Inspect how token representations change from isolated static embeddings to fully contextualized vectors.

Select a token to inspect LayerNorm and FFN transformation values.

Autoregressive Next-Token Simulator

The final token's contextualized vector is projected via Language Model head ($W_{vocab}$) to compute next-token probabilities.