Position Encoding Comparison › Extrapolation Benchmarks
Language AI HandbookAll chapters Position Encoding Comparison › Extrapolation BenchmarksOne section · Sections in this chapter3 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The most visible difference between position encoding methods appears when models encounter sequences longer than their training length. This scenario is increasingly common: models trained on 2K or 4K context windows must often process documents with 8K, 16K, or even longer sequences. How gracefully does each method handle this extrapolation?
Setting Up the Comparison
We'll simulate extrapolation by creating position encodings for a sequence twice as long as the "training" length and observing how the representations behave at unseen positions.
import numpy as np
# Configuration
train_length = 512 # Maximum position seen during training
test_length = 1024 # We'll evaluate up to this position
embed_dim = 64 # Dimension of position vectors
d_k = embed_dim # Dimension for queries/keysLet's implement each position encoding method and examine its behavior at extrapolated positions.
def sinusoidal_encoding(max_len, d_model):
"""
Generate sinusoidal position encodings.
Follows the original Transformer paper formula.
"""
positions = np.arange(max_len)[:, np.newaxis] # (max_len, 1)
dimensions = np.arange(d_model)[np.newaxis, :] # (1, d_model)
# Wavelengths from 2*pi to 10000*2*pi
wavelengths = 10000 ** (2 * (dimensions // 2) / d_model)
# Apply sin to even dimensions, cos to odd dimensions
encodings = np.where(
dimensions % 2 == 0,
np.sin(positions / wavelengths),
np.cos(positions / wavelengths),
)
return encodings
def learned_embedding(max_len, d_model, rng=None):
"""
Simulate learned position embeddings.
In practice, these would be learned during training.
"""
if rng is None:
rng = np.random.default_rng(42)
# Initialize with small random values (simulating learned embeddings)
return rng.standard_normal((max_len, d_model)) * 0.02def rope_encoding(positions, d_model):
"""
Compute RoPE rotation angles for given positions.
Returns the cos and sin components for rotation.
"""
# Frequencies decrease exponentially with dimension pair
freqs = 1.0 / (10000 ** (np.arange(0, d_model, 2) / d_model))
# Outer product: position * frequency
angles = positions[:, np.newaxis] * freqs[np.newaxis, :] # (len, d_model/2)
cos_vals = np.cos(angles)
sin_vals = np.sin(angles)
return cos_vals, sin_vals
def alibi_bias(query_len, key_len, num_heads=8):
"""
Compute ALiBi attention bias matrix.
Returns bias to be added to attention scores.
"""
# Slopes: geometric sequence from 2^(-8/n) to 2^(-8)
slopes = 2 ** (-8 * np.arange(1, num_heads + 1) / num_heads)
# Distance matrix: query_pos - key_pos
query_pos = np.arange(query_len)[:, np.newaxis] # (Q, 1)
key_pos = np.arange(key_len)[np.newaxis, :] # (1, K)
distances = query_pos - key_pos # (Q, K)
# Bias for each head: -slope * |distance|
# Shape: (num_heads, Q, K)
biases = -slopes[:, np.newaxis, np.newaxis] * np.abs(distances)
return biases, slopesMeasuring Representation Quality at Extrapolated Positions
For sinusoidal encoding, extrapolation is mathematically guaranteed to work because the formula applies to any position. For learned embeddings, we have no representation at all for unseen positions. For RoPE, the rotation angles continue following the same pattern. For ALiBi, the linear bias extends naturally.
# Generate encodings for both training and test lengths
sinusoidal_train = sinusoidal_encoding(train_length, embed_dim)
sinusoidal_test = sinusoidal_encoding(test_length, embed_dim)
learned_train = learned_embedding(train_length, embed_dim)
# Learned embeddings have NO representation for extrapolated positions
# We'd need to either: interpolate, extrapolate heuristically, or fail
# RoPE continues naturally
positions_train = np.arange(train_length)
positions_test = np.arange(test_length)
rope_cos_train, rope_sin_train = rope_encoding(positions_train, embed_dim)
rope_cos_test, rope_sin_test = rope_encoding(positions_test, embed_dim)
# ALiBi bias extends naturally
alibi_train, alibi_slopes = alibi_bias(train_length, train_length)
alibi_test, _ = alibi_bias(test_length, test_length)Before examining extrapolation, let's visualize how each method structures position information across the embedding dimensions. These heatmaps reveal the fundamental patterns each method uses to encode position.
The heatmaps reveal fundamental differences in how each method encodes position. Sinusoidal and RoPE show structured wave patterns with different frequencies across dimensions, enabling the model to capture both fine-grained (nearby positions) and coarse (distant positions) relationships. Learned embeddings start as random noise and must discover useful structure during training. ALiBi operates in a different space entirely, showing the triangular bias pattern that penalizes attention to distant keys.
Let's visualize how each method's representations look at both trained and extrapolated positions.
The extrapolation behavior differs dramatically between methods. Sinusoidal encoding continues its periodic patterns into the extrapolation region, with each frequency component oscillating according to its fixed formula. RoPE shows the same behavior: rotation angles continue according to the same exponentially-decaying frequency schedule. ALiBi's linear bias grows predictably, applying stronger penalties to more distant positions regardless of whether those distances were seen during training.
Learned embeddings, however, have a fundamental problem: they simply don't exist for unseen positions. The gray region in the plot represents a complete absence of positional information. Various workarounds exist (position interpolation, extrapolation via linear projection), but none match the clean mathematical extension of formula-based methods.
Quantifying Extrapolation Quality
Beyond visual inspection, we can measure extrapolation quality by examining whether the relative position relationships remain consistent. Good extrapolation should preserve the property that nearby positions have similar representations and that relative distances remain meaningful.
def compute_relative_similarity(encodings, query_pos, max_distance=100):
"""
Compute cosine similarity between a query position and nearby positions.
Returns similarities at each relative distance.
"""
query_vec = encodings[query_pos]
similarities = []
for dist in range(max_distance):
key_pos = query_pos + dist
if key_pos < len(encodings):
key_vec = encodings[key_pos]
# Cosine similarity
sim = np.dot(query_vec, key_vec) / (
np.linalg.norm(query_vec) * np.linalg.norm(key_vec) + 1e-8
)
similarities.append(sim)
else:
similarities.append(np.nan)
return np.array(similarities)
# Compare relative similarity patterns at trained vs extrapolated positions
trained_query = 256 # Well within training range
extrap_query = 768 # In extrapolation region
sinusoidal_sim_trained = compute_relative_similarity(
sinusoidal_test, trained_query
)
sinusoidal_sim_extrap = compute_relative_similarity(
sinusoidal_test, extrap_query
)
The nearly identical curves show that sinusoidal encoding preserves relative position relationships during extrapolation. Position 768 "looks at" its neighbors in the same way that position 256 does. This consistency is what allows models with sinusoidal or RoPE encoding to generalize, with appropriate scaling adjustments, to longer sequences.
Position Similarity Matrices
Another way to understand position encoding quality is through pairwise similarity matrices. These show how similar any two positions are in the encoding space, revealing whether relative relationships are preserved.
The sinusoidal similarity matrix displays the characteristic banded structure where similarity depends on the relative distance between positions, not their absolute values. This Toeplitz-like property (constant values along diagonals) is what enables position-invariant pattern learning. The learned embeddings, in contrast, show near-zero similarity everywhere except the diagonal. This reflects their random initialization. During training, learned embeddings would develop task-specific similarity structure.