Position Encoding Comparison › Position Encoding for Long Context
Language AI HandbookAll chapters Position Encoding Comparison › Position Encoding for Long ContextOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
As language models tackle increasingly long documents, position encoding becomes a bottleneck. Training on 2K tokens and inferring on 100K tokens requires extrapolation strategies that differ by method.
Context Length Scaling Strategies
Each position encoding method has different options for extending context length beyond training.
def rope_position_interpolation(target_len, train_len, base_freq=10000):
"""
RoPE with position interpolation for extended context.
Scales positions to fit within the training range.
"""
# Instead of using positions 0, 1, 2, ..., target_len-1
# We use 0, train_len/target_len, 2*train_len/target_len, ...
scale = train_len / target_len
positions = np.arange(target_len) * scale
return positions
def rope_ntk_interpolation(target_len, train_len, base_freq=10000, alpha=2.0):
"""
RoPE with NTK-aware interpolation.
Modifies the frequency base rather than scaling positions.
"""
# Scale the base frequency
new_base = base_freq * (alpha ** (target_len / train_len - 1))
return new_base
# Example: extending 4K training to 16K inference
train_len = 4096
target_len = 16384
linear_interp_positions = rope_position_interpolation(target_len, train_len)
ntk_base = rope_ntk_interpolation(target_len, train_len, alpha=2.0)
Different models use different extension strategies:
- Linear position interpolation (used in Code Llama): Scales all positions by the extension factor, so position 8192 in an 8K→16K extension becomes effective position 4096
- NTK-aware interpolation: Modifies the frequency base, preserving high-frequency patterns better
- YaRN (Yet another RoPE extension): Combines both approaches with frequency-dependent scaling
- ALiBi: Naturally extends because the linear bias formula applies to any distance
Comparing Long-Context Performance
Let's simulate how well each method maintains coherent attention patterns at extended context lengths.
def simulate_long_context_attention(
method, context_len, train_len, embed_dim=64
):
"""
Simulate attention pattern quality at different context lengths.
Returns a score representing pattern coherence (higher is better).
"""
if method == "learned":
# Learned embeddings have no extrapolation - severe degradation
if context_len > train_len:
return 0.1 + 0.1 * (train_len / context_len)
else:
return 1.0
elif method == "sinusoidal":
# Sinusoidal extrapolates mathematically but may drift
ratio = context_len / train_len
if ratio <= 1:
return 1.0
else:
# Gradual degradation due to unseen position combinations
return max(0.3, 1.0 - 0.15 * np.log2(ratio))
elif method == "rope":
# RoPE with position interpolation maintains quality better
ratio = context_len / train_len
if ratio <= 1:
return 1.0
else:
# With proper interpolation, degradation is slower
return max(0.5, 1.0 - 0.1 * np.log2(ratio))
elif method == "alibi":
# ALiBi designed for extrapolation
ratio = context_len / train_len
if ratio <= 1:
return 1.0
else:
# Best extrapolation due to simple linear design
return max(0.6, 1.0 - 0.05 * np.log2(ratio))
return 0.5
# Evaluate across context lengths
context_lengths = [512, 1024, 2048, 4096, 8192, 16384, 32768]
train_length = 2048
methods = ["learned", "sinusoidal", "rope", "alibi"]
results = {method: [] for method in methods}
for ctx_len in context_lengths:
for method in methods:
score = simulate_long_context_attention(method, ctx_len, train_length)
results[method].append(score)
The simulation reflects patterns observed in practice. ALiBi was specifically designed for length extrapolation, and its simple linear penalty on distance generalizes naturally. RoPE with appropriate interpolation techniques can extend well beyond training length, though some quality degradation occurs. Sinusoidal encoding extrapolates mathematically but may not maintain the attention patterns the model learned during training. Learned embeddings fail catastrophically beyond their defined range.