Deep Learning Architecture

How Self-Attention Works

Transformers understand context by computing how much each word (Token) should attend to every other word using Query (Q), Key (K), and Value (V) vectors.

1. Example Sentence

2. Focus Token (Query)

3. Attention Head Goal

Attention Weights Distribution

Computing attention weights from Query token to all Key tokens

Click any token to inspect its Query viewpoint:

Softmax Normalized Attention Scores ($Softmax(\frac{Q \cdot K^T}{\sqrt{d_k}})$)

1. Dot-Product Similarity

Multiplies the active Query vector with each token's Key vector to measure raw compatibility.

2. Scaling & Softmax

Divides by $\sqrt{d_k}$ to stabilize gradients, then exponentiates so all weights sum exactly to 100%.

3. Weighted Value Sum

Constructs the new contextualized token representation by blending all Value vectors by these scores.

Native head and query actions in an authored attention teaching tool

Your browser does not support video.

Read the explanation

This interactive tool calculates one query row using authored scoring rules. It does not compute learned query, key or value vectors. Hold the first native sentence and it query at source index seven. Change the actual head selector from coreference to syntax. For coreference, the assigned raw scores for animal and tired are eight point five and five point five. For syntax with this query, animal uses the distance fallback score one point six and tired three point two. The code exponentiates every raw score after division by the fixed number one point five and normalizes over all ten keys. Animal to tired therefore changes from a weight ratio of about seven point three eight nine to zero point three four four one five four. Both complete held rows have individual source-index labels. Every explainer bar uses three hundred forty pixels per weight unit. The original widget makes very small displayed bars at least two percent wide for visibility. These scientific explainer bars deliberately use exact proportional widths instead and label tiny probabilities honestly. No numerical reveal from zero pretends to be a source operation. This comparison explains a genuine head-selection action, not a trained coreference result. Keep the first sentence and it query. Now change the actual head selector from coreference to neighbor. The neighbor rule starts every raw score at one and adds the larger of zero point two or six minus one point five times the absolute key-query distance. The it key has distance zero and score seven. Tired is two positions away and has score four. Animal is six positions away and reaches the floor score one point two. These scores enter the same fixed-divisor exponentiation and normalization over all ten keys. Animal to tired now has a ratio of about zero point one five four six three eight, compared with seven point three eight nine in coreference. The two exact held rows show how a source index-distance rule changes the distribution. There is no temperature slider in this page and no learned positional head. The original two-percent visual minimum remains in the public widget; these explanatory bars use their full correctly scaled mathematical probabilities. Each row sums to one. The names are teaching goals assigned to heuristic rules, not measured transformer behavior or a computed value-vector output. Hold the first sentence and coreference head. Use the native query menu or token buttons to change the query from it, index seven, to tired, index nine. The tired-query branch assigns raw score seven to both animal and it, and two point five to every other key. Its full normalized row therefore gives animal and it equal weights. The it-query branch instead uses its assigned animal, street and tired boosts plus the distance fallback for the other keys. Both complete distributions are held here on one exact shared scale with every source token identified. The capitalized The and lowercase the have separate source positions even though their cleaned words are equal. The source exports a textual summary through its Copy Summary button; that summary does not contain a computed value vector or a trained model prediction. Native copying and its real feedback timer are preserved and tested. The page is an authored heuristic teaching tool. The explanation follows actual query-selection behavior without claiming learned semantics, training, a full attention matrix or hidden-state mixing.