Inside the Transformer: Visual Architecture & Attention Lab

1. Linguistic Corpus Preset

2. Attention Head (Specialization)

3. Softmax Temp (τ = 1.0)

0.2 (Sharp/Argmax) 2.5 (Uniform/Entropy)

Token Sequence & Active Query Vector (Q_i)

Click any token to inspect its attention distribution

Attention Weights: Softmax(Q · Kᵀ / √d_k)

Querying: "it" (Token #7)

Total Weight Σ: 100.0%

Projection & Next-Token Logits FFN + Unembed

Top candidate completions after contextual aggregation:

Mathematical Operation

Attention(Q, K, V) = Softmax(Q·Kᵀ / √d_k) · V

• Scaled by 1/√64 = 0.125 to avoid saturated gradients in large dimensions.
• Temperature τ = 1.0 smooths logit peaks before the final value aggregation.

Transformer Layer Lifecycle

1. Embed & Position

Discrete token IDs map to continuous d-dimensional vectors and combine with sinusoidal/RoPE positional signals.

2. QKV Linear Projections

Input matrix X multiplies projection weights W_q, W_k, W_v across h parallel attention heads.

3. Residual + LayerNorm

X + MHA(X) preserves uninterrupted gradient highways across dozens of stacked transformer blocks.

4. MLP / Feed-Forward

Position-wise two-layer expansion (typically 4x d_model) stores factual and linguistic knowledge patterns.

Native temperature, head and query rotation in an assigned attention lab

Your browser does not support video.

Read the explanation

This educational lab starts with hand-authored head weight vectors. Its JavaScript does not compute learned query, key or value vectors or run a trained transformer. Keep the first preset, semantic head, and it query at index seven. The source base weight for animal is zero point six five and for it zero point one zero. At temperature one, these assigned weights already sum to one, so normalization preserves them. Move the genuine temperature slider to two point five. The implementation clamps each base weight to at least zero point zero zero one, takes its logarithm, divides by temperature, subtracts the maximum logit, exponentiates and normalizes. Equivalently, these positive weights are raised to the power one over temperature and normalized. The animal to it ratio changes from six point five to about two point one one four three. Both complete eleven-key rows are shown at their exact values, using three hundred forty pixels per normalized weight unit. No numerical bar grows falsely from zero. The change follows the actual native input and assigned weights, without demonstrating learned language understanding or a hidden-state output. Keep the first preset, it query and temperature one. Choose the actual syntactic head instead of semantic. The semantic base vector assigns animal zero point six five and was zero point zero two. The syntactic base vector assigns animal zero point zero four and was zero point five two. Each vector is a stored teaching example, not a matrix learned during this session. Because the selected query is the preset default and temperature is one, no query rotation occurs and each normalized row equals its assigned base weights. The full eleven-key distributions show the head action directly, with every token index and probability labeled. The source page also displays a next-token prediction list, but that list is fixed for each preset. Its to, and and so entries do not change with this head selection, query change or temperature. Those displayed percentages do not come from the attention computation and are not model inference. We keep that original source behavior intact. This comparison teaches a change of assigned head vector, not a trained semantic or syntactic decision. Keep the first preset, semantic head and temperature one. Click animal, index one, instead of the default it query, index seven. The actual source does not calculate new semantic affinities for animal. It circularly shifts the stored vector by the selected query index minus the preset default index. Here that shift is negative six. The original zero point six five at index one moves to index six, because. The original zero point one zero at index seven moves to the new self index one, animal. All other entries rotate by the same rule before normalization. The two held eleven-key rows make this mechanical query action visible at one exact shared scale. Their sums remain one. A large because bar here is a consequence of circular rotation, not evidence that a language model chose because as the meaning of animal. The next-token list remains the same assigned preset list. Reset restores the first preset, semantic head, it query and temperature one. We preserve every original control, all twenty-four slider positions, native mouse and touch behavior and responsive tool dimensions. No learned vector mixing, model prediction or trained transformer output is claimed.