Why do Transformers need Positional Encodings?
Standard Self-Attention Softmax(QKᵀ / √d)V is completely permutation-equivariant: swapping tokens in the input yields the exact same outputs, just permuted. Without explicit positional vectors injected into embeddings, sentences like "Dog bites man" and "Man bites dog" produce identical attention weights.
Frequencies decay geometrically ω_k = 1 / 10000^(2k/d) allowing the model to attend by relative distances.
Dot products between positions decay smoothly as offset |i - j| increases, establishing inductive locality bias.
Why Self-Attention Cannot Distinguish Word Order
In an unaugmented self-attention layer, the representation of token x_i interacting with token x_j is computed solely via their inner product: q_i · k_j = (x_i W_Q) · (x_j W_K). Notice there is no index parameter i or j in this formula.
1. Permutation Equivariance
If you apply permutation matrix P to input sequence X, the layer computes Attention(PX) = P · Attention(X). The transformer processes an unordered multiset (bag of tokens).
2. Absolute vs Relative Positional Encoding
Absolute (Sinusoidal / Learned) adds vector x'_i = x_i + p_i. Relative / RoPE multiplies representations with rotation matrix R_Θ,m such that ⟨R_m q, R_n k⟩ = g(q, k, m-n).
3. Why not use Recurrence (RNN/LSTM)?
RNNs process sequential state h_t = f(h_{t-1}, x_t), which encodes order naturally but prevents O(1) parallelization and degrades over long context horizons.