Rotary Position Embedding (RoPE) › RoPE Frequency Patterns
Language AI HandbookAll chapters Rotary Position Embedding (RoPE) › RoPE Frequency PatternsOne section · Sections in this chapter13 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The frequency choices determine which relative offsets each dimension pair can distinguish. Let's visualize how the multi-frequency structure creates unique position signatures:
# Visualize how position affects embedding components
d_model = 64
max_pos = 100
frequencies = compute_rope_frequencies(d_model)
# For each position, compute the rotation for each dimension pair
# We'll track cos(pos * freq) and sin(pos * freq)
cos_components = np.zeros((max_pos, d_model // 2))
sin_components = np.zeros((max_pos, d_model // 2))
for pos in range(max_pos):
cos_components[pos] = np.cos(pos * frequencies)
sin_components[pos] = np.sin(pos * frequencies)
The heatmaps reveal the multi-scale nature of RoPE. Low-index dimension pairs (top rows) cycle rapidly, distinguishing nearby positions. High-index pairs (bottom rows) change slowly. This provides a coarse position signal. This structure resembles sinusoidal position encodings since both use similar frequency patterns. The key difference is that sinusoidal encodings add position information to embeddings, while RoPE rotates the embeddings themselves.
Every position in the sequence maps to a unique column pattern across all the dimension pairs. No two positions within the context window produce the same combination of cosine and sine values (unless positions are so far apart that even the slowest frequency has repeated, which requires distances well beyond any practical context). This uniqueness is what allows the model to discriminate between any two positions in the sequence.
We can also visualize the effective rotation that accumulates over a long sequence by tracking a single unit vector in each pair as position increases. The resulting picture shows clearly why high-frequency pairs lose discriminative power at large distances while low-frequency pairs preserve it:
