Interactive Deep Learning

How Do Transformers Work?

Transformers process language through Scaled Dot-Product Self-Attention. Each word creates Query (Q), Key (K), and Value (V) vectors to calculate contextual attention scores across every other word simultaneously.

1. Sample Sequence

2. Focus Token (Query)

3. Softmax Temperature 1.0×

Attention Weight Matrix (Q × Kᵀ)

Click any token to focus

Mathematical Breakdown

Attention Formula: Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ)·V

Query token: "it"

Highest attention: "animal" (84.2%)

Semantic resolution: Coreference link resolved through learned contextual embeddings.

Why Self-Attention Works

Unlike Recurrent Neural Networks (RNNs) that process word-by-word sequentially, Transformers process all tokens simultaneously in O(1) sequential operations, solving long-range context degradation.

1. Positional Encoding

Since the model processes tokens concurrently, sinusoidal vectors or learned embeddings are injected so the model knows word order.

2. Multi-Head Attention

Multiple attention projections run in parallel, enabling the network to jointly attend to syntax, parts of speech, and factual references.

3. Feed-Forward & Residuals

Contextual outputs pass through non-linear MLP layers with layer normalization and skip connections to maintain gradient stability.

Toy affinities become normalized attention

Your browser does not support video.

Read the explanation

For source query it, animal has authored score eight point four and tired has four point one. Bars share forty pixels per raw affinity. The teaching tool also assigns missing non-self affinities random scores between point two and point six. These are illustrative authored and sampled affinities, not learned query-key vectors or outputs of a trained transformer. The source softmax divides scores by temperature. For animal eight point four versus tired four point one, their weight ratio is exponential of four point three over temperature. It is about seventy-three point seven at temperature one and five point five eight four five at two point five. Bars share four pixels per ratio unit. Higher temperature reduces this preference; other keys change absolute probabilities while this relative ratio follows the same arithmetic. The source sample sentences include six and eleven tokens. A full dense attention matrix at those lengths would contain thirty-six and one hundred twenty-one query-key pairs. Bars share three pixels per pair. Source presets, every query/token control, temperature endpoints, reset and complete SVG/weights/explanation are compared using the same fresh random input in isolated headless tests, while public source randomness remains untouched. Native resize, first-footer and actual public playback checks still apply; independent scene approval is separate.