Transformer Architecture & Self-Attention Explorer

1. Choose Input Sequence

2. Attention Head Role

3. Softmax Temperature (1.0)

Select Focus Token (Query $Q$)

Click any token to compute its self-attention distribution

Attention Weight Matrix ($A = \text{softmax}(QK^T / \sqrt{d_k})$)

Mathematical Mechanism

Query (Q): What this token is looking for.

Key (K): What identity other tokens offer.

Value (V): The content transferred forward.

$\text{Score}(i,j) = \frac{q_i \cdot k_j}{\sqrt{d_k}}$ → $\alpha_{ij} = \frac{e^{\text{Score}_{ij}/T}}{\sum e^{\text{Score}_{in}/T}}$

Transformer Insight

1. Parallel Processing

Unlike Recurrent Neural Networks (RNNs) that parse words sequentially, Transformers digest all tokens in a sequence simultaneously, enabling massive parallelization on modern GPU hardware.

2. Scaled Dot-Product Attention

Tokens exchange information by projecting into Query, Key, and Value vectors. The dot product between Query and Key determines routing weights for pulling features from Values.

3. Multi-Head Representation

Multiple attention heads run in parallel across distinct subspace projections, simultaneously tracking grammatical rules, entity resolutions, and long-range semantic relationships.

Native temperature, head and query actions in an authored attention explorer

Your browser does not support video.

Read the explanation

This page is an authored attention teaching example. Its labels mention query, key and value, but the actual JavaScript assigns scalar scores rather than computing learned vectors or a trained transformer. Select the first sentence, it at source index seven, and the syntax head. Its eleven keys include because at index six and was at eight. These immediate neighbors both receive score two point two. Street at five and too at nine are two positions away and receive zero point eight. The query itself receives zero point two five before any other rule. The actual temperature slider moves from one to two. Every score is divided by temperature, exponentiated, and normalized over all eleven keys. The ratio of because to street changes from the exponential of one point four, about four point zero five five two, to the exponential of zero point seven, about two point zero one three eight. The two complete distributions remain held at their exact cases. Each bar uses three hundred forty pixels per normalized weight unit and includes a readable numeric label. This temperature action changes the source normalization; it does not prove language understanding or compute a value-vector output. Keep the first sentence, the it query and temperature one. Change the actual head menu from coreference to syntax. In the coreference source map, animal is assigned zero point seven two times four, or two point eight eight. Street is zero point zero eight times four, or zero point three two. Tired is zero point one two times four, or zero point four eight. These hand-authored rules take priority over fallback scoring. The query itself still receives zero point two five. Switching to syntax selects a different map and the actual adjacency rules. Because and was now receive two point two, while street and too receive zero point eight. The source fallback for remaining entries is zero point one five plus zero point one times one plus the sine of three times key index plus query index. All eleven entries take part in the same exponential normalization. Both exact source rows are shown on the shared scale, with indices distinguishing The from the lowercase the. A menu change causes a different assigned numerical distribution. The page does not train heads or compute an entire learned attention matrix; these are complete rows for the currently selected query. Keep the first sentence, the coreference head and temperature one. Click the actual animal token, index one, instead of it, index seven. The source coreference map has a row for it, but no explicit row for animal. The new query therefore uses the real fallback for every nonself key: zero point one five plus zero point one times one plus the sine of three times key index plus query index. The new self key, animal, receives the fixed score zero point two five. This is not a semantic inference about animal. It is a deterministic assigned fallback in this teaching page. Compare the complete eleven-key distributions, each with sum one, rather than drawing a hypothetical two-key softmax or inventing a new sentence option. Every numeric bar is held at its actual computed width throughout the explanation. Tiny probabilities retain the same scale as large probabilities. The native reset restores the first sentence, it query, coreference head and temperature one. Original mouse and touch token selection, all nineteen slider positions and responsive tool dimensions are preserved. No value-vector mixing, final hidden state or model prediction is produced by this source.