Context Length Challenges › Position Encoding Extrapolation
Language AI HandbookAll chapters Context Length Challenges › Position Encoding ExtrapolationOne section · Sections in this chapter4 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Position encodings tell the model where each token sits in the sequence. Without them, attention treats "The cat sat on the mat" and "mat the on sat cat The" identically. The challenge: most position encoding schemes fail catastrophically when asked to encode positions beyond their training range.
Absolute Position Embeddings
The original transformer used learned absolute position embeddings: a lookup table mapping each position index to a learned vector. If you train with positions 0 through 511, the embedding table has 512 entries. What happens at position 512? The model has never seen it.
import torch.nn as nn
class AbsolutePositionEmbedding(nn.Module):
"""Learned absolute position embeddings."""
def __init__(self, max_length, d_model):
super().__init__()
self.embeddings = nn.Embedding(max_length, d_model)
self.max_length = max_length
def forward(self, positions):
# What if positions > max_length - 1?
if positions.max() >= self.max_length:
raise ValueError(
f"Position {positions.max()} exceeds max {self.max_length - 1}"
)
return self.embeddings(positions)
# Example: BERT-style embedding
pos_embed = AbsolutePositionEmbedding(max_length=512, d_model=768)Absolute Position Embedding: Maximum position: 511 Embedding dimension: 768 Total parameters: 393,216 Position 511: ✓ (within range) Position 512: ✗ (Position 512 exceeds max 511)
Absolute embeddings have zero ability to extrapolate. The model either crashes or produces garbage for unseen positions. This hard limit explains why early BERT applications required truncating documents to 512 tokens.
Sinusoidal Position Encodings
The original "Attention Is All You Need" paper proposed sinusoidal encodings that compute position vectors using trigonometric functions rather than learning them from data. The key idea is to create position-dependent patterns at multiple frequencies, allowing the model to detect both local and distant positional relationships.
For a position in the sequence and dimension index in the encoding vector:
where:
- : the encoding value at position for even dimension
- : the encoding value at position for odd dimension
- : the absolute position in the sequence (0, 1, 2, ...)
- : the dimension index within the encoding (ranges from 0 to )
- : the total dimensionality of the position encoding (typically matches the model's hidden dimension)
- : a base constant that determines the range of wavelengths
The division by creates different frequencies across dimensions. Low-indexed dimensions (small ) oscillate rapidly with position, while high-indexed dimensions oscillate slowly. This multi-scale representation allows the model to capture both fine-grained local position (from high-frequency components) and coarse global position (from low-frequency components).
Since these are deterministic mathematical functions rather than learned embeddings, they can generate vectors for any position, including positions never seen during training. But can the model use them beyond its training range?
def sinusoidal_encoding(positions, d_model):
"""Compute sinusoidal position encodings."""
positions = np.array(positions).reshape(-1, 1)
dims = np.arange(d_model).reshape(1, -1)
# Compute the divisor term
div_term = 10000 ** (2 * (dims // 2) / d_model)
# Apply sin to even indices, cos to odd
encodings = np.zeros((len(positions), d_model))
encodings[:, 0::2] = np.sin(positions / div_term[:, 0::2])
encodings[:, 1::2] = np.cos(positions / div_term[:, 1::2])
return encodings
# Generate encodings for positions within and beyond training
d_model = 64
train_positions = np.arange(512)
extrapolate_positions = np.arange(512, 2048)
train_encodings = sinusoidal_encoding(train_positions, d_model)
extrap_encodings = sinusoidal_encoding(extrapolate_positions, d_model)
Sinusoidal encodings produce mathematically valid vectors at any position. The issue is semantic: the model learned to interpret positions 0-511 during training. Position 1024 might have a perfectly valid encoding, but the model hasn't learned what attention patterns make sense when tokens are that far apart.
Rotary Position Embeddings and the Extrapolation Problem
Rotary Position Embeddings (RoPE), used in LLaMA and many modern models, encode relative positions through rotation in embedding space. They offer better extrapolation than absolute embeddings but still degrade beyond training length.
def rope_extrapolation_analysis(train_length, test_lengths, d_model=64):
"""
Analyze RoPE behavior at different positions.
RoPE encodes relative position through rotation angle:
theta = position * base^(-2i/d)
At long positions, high-frequency components rotate very fast,
creating patterns the model hasn't seen.
"""
base = 10000
dims = np.arange(0, d_model, 2)
results = []
for pos in test_lengths:
# Compute rotation frequencies at this position
thetas = pos / (base ** (dims / d_model))
# High-frequency dimensions (large dim index) rotate slowly
# Low-frequency dimensions (small dim index) rotate faster
max_rotation = thetas.max()
avg_rotation = thetas.mean()
# How many full rotations in highest frequency?
high_freq_rotations = thetas[0] / (2 * np.pi)
# Beyond training, high frequencies enter unseen territory
in_training = pos <= train_length
results.append(
{
"position": pos,
"max_theta": max_rotation,
"high_freq_rotations": high_freq_rotations,
"in_training": in_training,
}
)
return results
train_len = 2048
test_positions = [512, 1024, 2048, 4096, 8192, 16384]
rope_analysis = rope_extrapolation_analysis(train_len, test_positions)RoPE Rotation Analysis (trained on 2048 tokens):
Position Max θ High-freq Rotations Status
------------------------------------------------------------
512 512.00 81.5 In training
1,024 1024.00 163.0 In training
2,048 2048.00 325.9 In training
4,096 4096.00 651.9 EXTRAPOLATION
8,192 8192.00 1303.8 EXTRAPOLATION
16,384 16384.00 2607.6 EXTRAPOLATIONAt position 4096 (2x training length), high-frequency components have rotated 163 times. At position 16384 (8x training length), they've rotated 653 times. The model has never seen these rotation patterns. The mismatch manifests as degraded attention: tokens that should attend to each other might not, and irrelevant tokens might receive spurious attention.
The visualization shows why position interpolation (covered in the next chapter) works: instead of asking the model to interpret new rotation regimes, we rescale positions to stay within the familiar training range. Position 4096 becomes position 2048, which the model understands.