RoPE (Su et al., 2021) encodes relative token positions by breaking query and key vectors into 2D orthogonal pairs and multiplying each pair by a 2D rotation matrix \(R_{\Theta, m}^d\). The inner product \(\langle R_m q, R_n k \rangle\) depends solely on relative distance \((m - n)\).
Query \(\mathbf{q}\) components \([q_0, q_1, q_2, q_3]\):
Key \(\mathbf{k}\) components \([k_0, k_1, k_2, k_3]\):
As token distance \(|m - n|\) increases, the multi-frequency rotary components oscillate and naturally decay the expected inner product \(\mathbb{E}[\langle R_m \mathbf{q}, R_n \mathbf{k} \rangle]\), providing Transformers with natural locality bias without static position bias tables.
This saved teaching tool rotates two coordinate pairs, using four input components. Its head dimension slider sets the frequency formula; it does not create sixty four input components or run a language model. For the default query pair one and point five at position two, the first frequency is one radian per position. Rotating through two radians produces approximately negative point eight seven zero eight and point seven zero one two. Cyan shows the original vector and amber the rotated vector. Both drawings use one hundred sixty pixels per coordinate unit and the same origin. The rotation preserves the length, approximately one point one one eight, while changing the direction. The second pair uses a slower frequency. At base ten thousand and head dimension sixty four, theta one is ten thousand raised to negative two over sixty four, approximately point seven four nine eight nine four radians per position. Query position two rotates its second pair point eight and negative point four through about one point four nine nine eight radians. The result is approximately point four five five eight and point seven six nine six. This drawing keeps the same one hundred sixty pixels per coordinate unit. The second pair still has length approximately point eight nine four four. Each independent two dimensional rotation preserves its own length. These are the first two frequency planes, not a complete sixty four dimensional embedding. The useful identity concerns the dot product. Rotating the query by position m and the key by position n gives the same result as keeping the query fixed and rotating the key by n minus m. With this tool’s default four components and positions two and one, the raw dot product is point nine two. The two rotated vectors have dot product about one point two two three nine eight nine. The relative shift calculation gives the same value up to floating point rounding, with residual about two point two times ten to negative sixteen. A shared absolute shift cancels because the rotation matrix transpose times itself is the identity. That algebra proves relative position dependence; it does not prove empirical model quality or improved performance over other embeddings.