How Transformers Work: Self-Attention

Interactive Mechanism

Transformers replace sequential loops with Self-Attention: every token computes a Query ($Q$), Key ($K$), and Value ($V$) to weigh contextual relevance with every other token in parallel.

Example Sentence

Attention Head Perspective

Softmax Temperature: 1.0

1. Select Query Token ($Q$) to Inspect Attention ($QK^T / \sqrt{d_k}$)

Click any token badge

Query Word: it Thickness & Opacity = Attention Weight ($W_{ij}$)

Normalized Attention Weights (Softmax)

Attention Math Breakdown

For token it, the model computes compatibility scores against all Keys ($K$), scales by dimension $\sqrt{d}$, and applies Softmax:

$\text{Score}(Q, K_i) = Q \cdot K_i^T / \sqrt{d_k}$

$\alpha_i = \exp(\text{Score}_i / T) / \sum_j \exp(\text{Score}_j / T)$

$\text{Output Context} = \sum \alpha_i V_i$

Resolving reference...