Interactive Attention Map
Head 1: Syntactic Relations
Tokens in Context ($X \in \mathbb{R}^{n \times d}$)
Scaled Dot-Product Attention: $\text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)$
Weights $A \in [0, 1]$
Multi-Head Attention Layers (MHA)
Head 1
dim 64
Syntactic & modifier dependencies (adjectives & nouns).
Head 2
dim 64
Coreference & entity resolution ("it" → antecedent).
Head 3
dim 64
Local positional context (adjacent n-gram tokens).
Head 4
dim 64
Semantic verb-argument structure (actions & subjects).
Tensors & Mathematical Projections
$d_{model} = 64$
$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$
For query token "it":
dot product with keys determines context aggregation from values $V$.
Attention Distribution for Selected Token ($A_{i,:}$)
Query ($q_i$) & Top Key ($k_j$) Dot Product Decomposition
Context Vector: $z_i = \sum_j A_{i,j} v_j$ (Value Weighted Sum)
Next Transformer Stage: Add & Norm $\rightarrow \text{LayerNorm}(x + \text{MHA}(x)) \rightarrow \text{FFN}(z) \rightarrow \text{LayerNorm}(z + \text{FFN}(z))$.