Interactive Deep Learning

How Do Transformers Work?

Transformers process language through Scaled Dot-Product Self-Attention. Each word creates Query (Q), Key (K), and Value (V) vectors to calculate contextual attention scores across every other word simultaneously.

1. Sample Sequence

2. Focus Token (Query)

3. Softmax Temperature 1.0×

Attention Weight Matrix (Q × Kᵀ)

Click any token to focus

Mathematical Breakdown

Attention Formula: Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ)·V

Query token: "it"

Highest attention: "animal" (84.2%)

Semantic resolution: Coreference link resolved through learned contextual embeddings.

Why Self-Attention Works

Unlike Recurrent Neural Networks (RNNs) that process word-by-word sequentially, Transformers process all tokens simultaneously in O(1) sequential operations, solving long-range context degradation.

1. Positional Encoding

Since the model processes tokens concurrently, sinusoidal vectors or learned embeddings are injected so the model knows word order.

2. Multi-Head Attention

Multiple attention projections run in parallel, enabling the network to jointly attend to syntax, parts of speech, and factual references.

3. Feed-Forward & Residuals

Contextual outputs pass through non-linear MLP layers with layer normalization and skip connections to maintain gradient stability.