Deep Learning Architecture

How Self-Attention Works

Transformers understand context by computing how much each word (Token) should attend to every other word using Query (Q), Key (K), and Value (V) vectors.

1. Example Sentence

2. Focus Token (Query)

3. Attention Head Goal

Attention Weights Distribution

Computing attention weights from Query token to all Key tokens

Click any token to inspect its Query viewpoint:

Softmax Normalized Attention Scores ($Softmax(\frac{Q \cdot K^T}{\sqrt{d_k}})$)

1. Dot-Product Similarity

Multiplies the active Query vector with each token's Key vector to measure raw compatibility.

2. Scaling & Softmax

Divides by $\sqrt{d_k}$ to stabilize gradients, then exponentiates so all weights sum exactly to 100%.

3. Weighted Value Sum

Constructs the new contextualized token representation by blending all Value vectors by these scores.