Transformer Architecture & Self-Attention Interactive Workbench

1. Scenario Presets

Language Understanding Benchmark

2. Attention Head dim = 64 / head

Head 1 (Coreference Tracking): High cross-attention between pronouns like "it" and candidate antecedents like "animal".

3. Mathematical Knobs

Softmax Temperature (τ) 1.00

Low τ sharpens focus (argmax-like); high τ diffuses weights uniformly.

Apply Causal Mask (Decoder-only GPT style)

Include Sinusoidal Positional Encoding

4. Autoregressive Step

Transformers output a logit distribution over vocabulary to sample the next token, then append it and repeat.

Input Sequence (Click any token to set as Active Query $q_i$) 9 Tokens

Step 1

Scaled Dot-Product Matrix

Step 2

Q × K × V Decomposition

Step 3

MLP & Residual Stream

Step 4

Next-Token Logits

Attention Heatmap: $\text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)$ Hover cell to inspect

Active Query Spotlight • Token "it"

Evaluating how much Query token $i$ attends to each Key token $j$ across the sequence.

$\text{Score}(i, j) = \frac{q_i \cdot k_j}{\sqrt{d_k}}$
$\alpha_{i,j} = \frac{\exp(\text{Score}_{i,j} / \tau)}{\sum_k \exp(\text{Score}_{i,k} / \tau)}$

Top Attended Keys from Query:

Context Value Aggregation:

$z_i = \sum_j \alpha_{i,j} \cdot v_j \longrightarrow \text{Weighted combination of antecedent embeddings}$

Linear Projections ($W_Q, W_K, W_V$)

Raw token embeddings $x_i \in \mathbb{R}^{d_{model}}$ are multiplied by learned weight matrices to produce specialized representations:

Query (Q): "What am I looking for?" Key (K): "What information do I offer?" Value (V): "What content do I transmit?"

Token Embedding Vector for "it":

Query Vector ($q = x \cdot W_Q$):

Key Vector ($k = x \cdot W_K$):

Value Vector ($v = x \cdot W_V$):

Geometric Vector Space (2D PCA Projection)

High dot product $q_i \cdot k_j$ occurs when vectors point in collinear directions with high magnitude.

Why Transformers Revolutionized AI: Core Invariants

• Constant Path Length $\mathcal{O}(1)$

Unlike Recurrent Neural Networks (RNNs) that process word-by-word with vanishing gradients across long distances, self-attention connects any two tokens in a single operation regardless of distance.

• Massively Parallel Matrix Math

All queries, keys, and values are computed simultaneously via batched matrix multiplication ($Q K^T$), fully saturating GPU tensor cores during training.

• Multi-Head Specialization

Multiple attention heads learn complementary linguistic roles: syntactic subject-verb dependencies, coreference disambiguation, positional offsets, and semantic broad context in parallel.

• Constant Path Length $\mathcal{O}(1)$

• Massively Parallel Matrix Math

• Multi-Head Specialization

Transformer State Export