Rotary Position Embedding (RoPE) › Relative Position Through Dot Products
Language AI HandbookAll chapters Rotary Position Embedding (RoPE) › Relative Position Through Dot ProductsOne section · Sections in this chapter10 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We've derived mathematically that RoPE should make attention scores depend only on relative position. Now let's verify this core property empirically and see what it looks like in practice.
The test is straightforward: create identical query and key vectors at different positions, apply RoPE, and compute attention scores. If RoPE works as intended, the scores should form a Toeplitz matrix, where each diagonal contains identical values. This structure proves that scores depend only on relative position (the difference between query and key positions), not on absolute positions.
Think of a Toeplitz matrix as a kind of shift-invariant fingerprint. In a standard matrix, the value at row , column can be anything. In a Toeplitz matrix, the value depends only on the offset . If you shift both row and column indices by the same amount, you get the same value. That is exactly what relative-position encoding produces: shifting the entire sequence by a constant does not change any attention score, because all relative distances remain the same.
def compute_rope_attention_scores(queries, keys, frequencies):
"""Compute attention scores with RoPE applied.
Args:
queries: Query vectors, shape (seq_len, d)
keys: Key vectors, shape (seq_len, d)
frequencies: RoPE frequencies, shape (d/2,)
Returns:
Attention scores, shape (seq_len, seq_len)
"""
seq_len, d = queries.shape
scores = np.zeros((seq_len, seq_len))
for m in range(seq_len):
# Rotate query at position m
q_rotated = rope_complex(queries[m], m, frequencies)
for n in range(seq_len):
# Rotate key at position n
k_rotated = rope_complex(keys[n], n, frequencies)
# Compute dot product
scores[m, n] = np.dot(q_rotated, k_rotated)
return scores
# Create test queries and keys
seq_len = 6
d_model = 8
# All queries are identical, all keys are identical
# This isolates the effect of position
q_template = np.random.randn(d_model)
k_template = np.random.randn(d_model)
queries = np.tile(q_template, (seq_len, 1))
keys = np.tile(k_template, (seq_len, 1))
frequencies = compute_rope_frequencies(d_model)
scores = compute_rope_attention_scores(queries, keys, frequencies)
The Toeplitz structure is clear: all entries along each diagonal are identical. Position (0,0), (1,1), (2,2) all have the same score (relative distance 0). Position (0,1), (1,2), (2,3) all match (relative distance 1). This is the relative position property in action.
Let's verify numerically by extracting scores for each relative distance:
# Extract scores by relative distance
relative_scores = {}
for m in range(seq_len):
for n in range(seq_len):
rel_dist = n - m
if rel_dist not in relative_scores:
relative_scores[rel_dist] = []
relative_scores[rel_dist].append(scores[m, n])Scores grouped by relative position (n - m): Relative Position Scores Std Dev --------------------------------------------------------------------------- -5 0.4769 0.00e+00 -4 0.1021, 0.1021 2.78e-16 -3 2.0972, 2.0972, 2.0972 0.00e+00 -2 4.4375, 4.4375, 4.4375, 4.4375 0.00e+00 -1 4.7684, 4.7684, 4.7684, 4.7684, ... 6.88e-16 0 2.5720, 2.5720, 2.5720, 2.5720, ... 4.80e-16 1 -0.3545, -0.3545, -0.3545, -0.3545, ... 3.10e-16 2 -1.5489, -1.5489, -1.5489, -1.5489 3.51e-16 3 -0.1455, -0.1455, -0.1455 1.89e-16 4 2.3313, 2.3313 3.14e-16 5 3.3711 0.00e+00
All standard deviations are effectively zero (within floating-point precision), confirming that scores at each relative distance are identical.
How Dot Products Vary with Relative Distance
The Toeplitz structure tells us scores depend only on relative position, but how do they vary? Let's trace how the dot product changes as we increase the relative distance between query and key:
# Analyze how dot product varies with relative distance
d_model = 32
frequencies = compute_rope_frequencies(d_model)
# Create random query and key vectors
q = np.random.randn(d_model)
k = np.random.randn(d_model)
# Compute dot products at various relative distances
max_rel_dist = 50
relative_distances = np.arange(-max_rel_dist, max_rel_dist + 1)
dot_products = []
# Fix query at position 50 (middle of range)
query_pos = 50
q_rotated = rope_complex(q, query_pos, frequencies)
for rel_dist in relative_distances:
key_pos = query_pos + rel_dist
k_rotated = rope_complex(k, key_pos, frequencies)
dot_products.append(np.dot(q_rotated, k_rotated))
dot_products = np.array(dot_products)
The oscillating pattern is characteristic of RoPE. The multiple frequencies create a complex interference pattern where some relative distances produce higher scores than others. This structure allows the model to learn position-dependent attention patterns during training.