Inside the Transformer: Visual & Mathematical Architecture Simulator

2. Scaled Dot-Product Attention Heatmap & Weights

$N \times N$ Matrix ($d_k=4$)

Y-axis: Query token ($i$) Cell color intensity = $\alpha_{ij} = \text{Softmax}$ Attention Weight X-axis: Key token ($j$)

3. Live Mathematical Arithmetic Inspector

q[it] × k[animal]

Step 1: Scaled Dot Product

Select a cell to view vector dot product

Step 2: Softmax Normalization across Row

$\alpha_{ij} = \exp(S_{ij}) / \sum_k \exp(S_{ik})$

Step 3: Weighted Context Vector $\mathbf{z}_i = \sum \alpha_{ij} \mathbf{v}_j$

Dim	Query $q_i$	Key $k_j$	Value $v_j$	Out $z_i$

Fabricated Vectors, Softmax and Causal Masks

Read the explanation

Inside the Transformer is an educational architecture lab, not a live model inference service. For the input it animal tired, head one constructs four dimensional query and key vectors from a token hash and a sine function, rounded to two decimal places. It adds hand written educational biases: the query it receives a coreference bias of three point two toward animal, and one point two toward itself. It divides the adjusted dot products by square root of four, which is two, then applies a numerically stable softmax. The displayed bars show the resulting probabilities on a single seven hundred pixel scale per probability one. These patterns demonstrate mechanics; they are not learned linguistic evidence or proof that a trained model reasons this way. Switch causal masking on with the same three tokens. For query position zero, only key position zero is allowed. Later animal and tired keys are assigned negative infinity scores and zero probability. Softmax over the only remaining key yields probability one on it. These bars use the same seven hundred pixel scale as before. Masking changes which inputs can contribute to context; it does not verify semantic truth. Click an individual cell to inspect source vectors, scaled scores and the normalized row. Every row has its own normalization, so compare values within the selected query rather than treating the entire matrix as one probability distribution. Turn the mask off and select the local adjacent head. The source changes the hash seed, so query and key vectors change. It also replaces the coreference rules with position biases: two point two for itself, two point eight for the next token, and one for a distance of two. The normalized weights for the same query therefore differ. Bars remain on seven hundred pixels per probability one. The source forms each context vector by summing probability times value vector, then rounds the four components to two decimal places for display. Presets, token selection and reset let you inspect this fabricated model reproducibly without any backend model request.