Attention Mechanism & Symmetry

Why Do Transformers Need Positional Encodings?

Self-attention computes token relationships using set operations ($QK^T$). Without positional encoding, it is permutation equivariant: reversing or shuffling word order produces identical token representations, erasing grammatical syntax.

1. Compare Ordered Sentences

2. Active Sentence Order

3. Positional Encoding (PE) Mode

None (Pure Self-Attention)

Permutation invariant bag-of-words. Token representations collide.

Sinusoidal PE ($+ \sin/\cos$)

Adds unique high-dimensional frequency vectors to break symmetry.

Order Aware

Self-Attention Weight Matrix ($A = \text{softmax}(QK^T / \sqrt{d_k})$)

PE: Active

Low Attention ($< 0.20$)

High Attention ($> 0.50$)

Meaning Preserved

Because positional vectors were added to word embeddings, the attention weights depend both on token semantics and relative positions. "Dog" attending to "bites" as subject yields a different vector than "Dog" as object.

Key Takeaway

Unlike Recurrent Neural Networks (RNNs) which process tokens step-by-step ($t_1 \to t_2 \to t_3$), Transformers process all tokens in parallel. A standard dot product $q_i \cdot k_j$ has no native sense of index $i$ or $j$. Adding positional encodings $x_i = e_i + p_i$ is what gives the model the concept of space and sequence.