Rotary Position Embedding (RoPE) Visualizer & Matrix Workbench

1. Token Positions Δ (m - n) = 1

Query Token Position (m) 2

Key Token Position (n) 1

2. RoPE Hyperparameters

Base Frequency (\(b\)) 10,000

Head Dimension (\(d\)) 64

3. Raw Query / Key Vectors

Query \(\mathbf{q}\) components \([q_0, q_1, q_2, q_3]\):

Key \(\mathbf{k}\) components \([k_0, k_1, k_2, k_3]\):

Relative Dot-Product Invariance & Proof \(\langle R_m q, R_n k \rangle = \langle q, R_{n-m} k \rangle\)

Raw Dot Product (No Pos)

2.0400

\(\mathbf{q}^T \mathbf{k}\) unrotated

RoPE Rotated Dot Product

1.9842

\((R_m \mathbf{q})^T (R_n \mathbf{k})\)

Shifted Relative Dot Product

1.9842

\(\mathbf{q}^T (R_{n-m} \mathbf{k})\)

Invariance Absolute Residual

0.000000

Exact machine tolerance \(\le 10^{-6}\)

2D Orthogonal Rotary Planes Plane 0 vs Plane 1

Plane 0 (\(i=0\)) \(\theta_0 = 1.0000\) rad

Fastest rotation (\(\theta_0 = b^{-0/d} = 1\))
Query angle: 0.0° | Key angle: 0.0°

Plane 1 (\(i=1\)) \(\theta_1 = 0.7499\) rad

Slower rotation (\(\theta_i = b^{-2i/d}\))
Query angle: 0.0° | Key angle: 0.0°

Query vector \(\mathbf{q}\) rotated by \(m\theta\)

Key vector \(\mathbf{k}\) rotated by \(n\theta\)

Unrotated base vectors

Attention Score / Dot Product vs Relative Distance \((m - n)\) Long-Term Decay Active

As token distance \(|m - n|\) increases, the multi-frequency rotary components oscillate and naturally decay the expected inner product \(\mathbb{E}[\langle R_m \mathbf{q}, R_n \mathbf{k} \rangle]\), providing Transformers with natural locality bias without static position bias tables.

Mathematical Formulation

1. 2D Rotary Matrix for dimension pair \(i\):
\[ R_{\Theta, m}^{(i)} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}, \quad \text{where } \theta_i = b^{-2i/d} \]
2. Orthogonality & Dot Product Preservation:
\[ (R_m \mathbf{q})^T (R_n \mathbf{k}) = \mathbf{q}^T R_m^T R_n \mathbf{k} = \mathbf{q}^T R_{n - m} \mathbf{k} = g(\mathbf{q}, \mathbf{k}, n - m) \]
3. Why RoPE outperforms Absolute Embeddings (Learned / Sinusoidal):
Because \(R_{\Theta, m}\) is an orthogonal matrix (\(R^T R = I\)), vector magnitudes are strictly preserved \(\|R_m \mathbf{q}\| = \|\mathbf{q}\|\). Relative position enters strictly as phase shifts in the complex frequency domain.