Neural Architecture

How Transformers Work

Transformers replace recurrence with Self-Attention, letting every token compare its Query (Q) against all Keys (K) simultaneously to compute attention weights and aggregate Values (V).

1. Choose Input Sequence

2. Attention Head Role

3. Softmax Temperature 1.0

Lower focuses intensely on top matches; higher diffuses weight across all tokens.

Self-Attention Equation

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V

Click any token tag to inspect which words it attends to most strongly.

Active Query Token (Q) Focus

Query: "it"

Select Query Token:

Attention Weights Distribution ($A_{i,j}$):

Attention Heatmap (All Tokens vs All Keys)

Core Transformer Pipeline Explained

1. Embed & Pos Encode

Words convert to numerical vectors + sinusoidal position tags to capture order without sequential RNN loops.

2. Multi-Head Attention

Linear projections yield Q, K, V. Parallel heads attend to syntax, co-reference, and long-range semantics simultaneously.

3. Residuals & Norm

Skip connections ($x + \text{Sublayer}(x)$) and LayerNorm stabilize deep gradients across 12 to 100+ stacked blocks.

4. Feed-Forward MLP

Position-wise 2-layer MLPs process attended representations, projecting knowledge before next layers or final output.