Explore how Multi-Head Self-Attention allows tokens to contextualize meaning across an entire sequence simultaneously.
Click any key token below to change focus or observe attention distribution:
Each token projects into a Query vector ($Q$) and Key vector ($K$). Their dot product measures mutual compatibility and semantic alignment across tokens.
Raw dot-products are divided by $\sqrt{d_k}$ to stabilize gradients, then normalized through Softmax so all attention weights across the sequence sum to 100%.
The token aggregates information by computing a weighted sum of all Value vectors ($V$), updating its contextual state without recurrent loops.