How Transformers Work

Explore the Self-Attention mechanism: how tokens query context across entire sequences simultaneously.

Input Sequence

Attention Head Focus

Step 1: Select Active Query Token ($Q$)

Click any token to compute its attention weights against all Key ($K$) tokens.

Formula: Softmax((Q × Kᵀ) / √dₖ) × V

Attention Flow Visualization

Showing weights projecting from "it" to all keys

Calculated Attention Scores

1. Query, Key, Value Vectors

Every token projection creates three vectors: Q (what I search for), K (what I contain), and V (the actual information payload to pass forward).

2. Scaled Dot-Product

The model multiplies $Q \cdot K^T$ to measure affinity. Scaling by $1/\sqrt{d_k}$ prevents vanishing gradients before applying Softmax to form probability distributions.

3. Parallel Processing

Unlike Recurrent Neural Networks (RNNs) that process step-by-step, Transformers compute all token relationships in parallel across multiple attention heads.

Attention normalizes scores into contextual weights

Your browser does not support video.

Read the explanation

The source positional head assigns one divided by distance plus one. A query at distance zero from itself gets score one, while an adjacent key at distance one gets point five. Bars share three hundred twenty pixels per raw score. These are heuristic scores, not normalized probabilities, and this educational tool does not run a trained transformer. The source exponentiates each heuristic score times two point five and divides by the sum. For a hypothetical two-key row with scores one and point five, normalized weights are about point seven seven seven three and point two two two seven. Bars share four hundred pixels per probability. Each row sums to one, so strengthening one score also changes the other key weight. A full dense self-attention layer with hypothetical five and ten tokens has twenty-five and one hundred query-key pairs. Bars share four pixels per pair. This source visualizer shows the selected query row rather than executing a full neural layer. All source sentence choices, head choices and token selections, reset, native responsive allocation and first-footer placement must remain intact; independent scene approval is separate.