Select Focus Token (Query $Q$)
Click any token to compute its self-attention distributionAttention Weight Matrix ($A = \text{softmax}(QK^T / \sqrt{d_k})$)
Mathematical Mechanism
Query (Q): What this token is looking for.
Key (K): What identity other tokens offer.
Value (V): The content transferred forward.
$\text{Score}(i,j) = \frac{q_i \cdot k_j}{\sqrt{d_k}}$ → $\alpha_{ij} = \frac{e^{\text{Score}_{ij}/T}}{\sum e^{\text{Score}_{in}/T}}$
Transformer Insight
1. Parallel Processing
Unlike Recurrent Neural Networks (RNNs) that parse words sequentially, Transformers digest all tokens in a sequence simultaneously, enabling massive parallelization on modern GPU hardware.
2. Scaled Dot-Product Attention
Tokens exchange information by projecting into Query, Key, and Value vectors. The dot product between Query and Key determines routing weights for pulling features from Values.
3. Multi-Head Representation
Multiple attention heads run in parallel across distinct subspace projections, simultaneously tracking grammatical rules, entity resolutions, and long-range semantic relationships.