Relative Position Encoding › Why Relative Position Matters

Michael BrenndoerferUpdated June 2, 20255 min read
Language AI HandbookAll chapters
Part 14 of 67Positional Encoding
Chapter 99Relative Position Encoding › Why Relative Position Matters
Reading section 2 of 16 · Why Relative Position Matters · 5 minSection 2 of 16 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 99 of 513

Section 2 of 16 · 5 min read

Relative Position Encoding › Why Relative Position MattersOne section · Sections in this chapter2 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Consider the sentence "The cat that I saw yesterday sat on the mat." The verb "sat" needs to find its subject "cat." With absolute position encoding, the model must learn that "position 5 attending to position 2" captures a subject-verb relationship. But if we move the same phrase later in the document, it becomes "position 105 attending to position 102." The model must learn these patterns for every possible absolute position pair, treating what is essentially the same linguistic relationship as hundreds of distinct cases.

The Generalization Problem

Absolute position encodings create position-specific patterns. A model trained on short sequences may never see "position 500 attending to position 497," even though the relationship is identical to "position 5 attending to position 2": a distance of 3 positions.

Relative position encoding solves this by focusing on the offset between positions. The distance from position 5 to position 2 is −3-3 (looking backward three positions). The distance from position 105 to position 102 is also −3-3. By encoding this offset rather than absolute positions, the model learns a single pattern for "three positions back" that generalizes across the entire sequence.

This shift has practical consequences that go beyond test-set metrics. Language has many distance-dependent patterns: adjectives typically appear one position before their nouns, determiners precede noun phrases by a few positions, and verbs often follow their subjects within a local window. These patterns are not anchored to absolute positions; they describe structural relationships that can appear anywhere in a document. Relative encoding captures these patterns directly, letting a single learned representation cover all occurrences of a given structural relationship regardless of where that relationship appears in the input.

The generalization benefit becomes especially important for long-context applications. A model trained on sequences of 512 tokens and then applied to sequences of 2048 tokens will encounter absolute positions it has never seen. Absolute encodings have no principled way to handle this: the embeddings for positions 513 through 2048 were never trained. Relative encodings, by contrast, are anchored to distances rather than indices. As long as the model has seen sequences where tokens are kk positions apart, it can apply that knowledge at any absolute location where that same distance appears.

In[3]:
Code
# Demonstrate the generalization problem with absolute positions
def count_position_pairs(max_len_train, max_len_test):
    """Count how many position pairs in test were never seen during training."""
    seen_pairs = set()
    for i in range(max_len_train):
        for j in range(max_len_train):
            seen_pairs.add((i, j))

    unseen = 0
    total = 0
    for i in range(max_len_test):
        for j in range(max_len_test):
            total += 1
            if (i, j) not in seen_pairs:
                unseen += 1

    return unseen, total


# Training on sequences up to length 128, testing on 256
unseen_abs, total_abs = count_position_pairs(128, 256)
Out[4]:
Console
Absolute Position Generalization Problem
=============================================
Training sequence length: 128
Test sequence length:     256
Total position pairs at test time: 65,536
Unseen position pairs:            49,152
Percentage unseen:                 75.0%

With relative encoding, a distance of -3 seen during training
applies equally to positions (5, 2) and (105, 102).
Out[5]:
Visualization
Heatmap showing seen vs unseen position pairs, with a dark square in the bottom-left corner and light L-shaped region indicating unseen pairs.
Visualization of the absolute position generalization problem. The dark region (bottom-left) represents position pairs seen during training on 128-token sequences. The light region shows unseen pairs when testing on 256 tokens. Nearly 75% of test pairs were never seen during training, meaning a model relying purely on absolute positions encounters mostly unfamiliar combinations at inference time.

Nearly 75% of position pairs at test time were never seen during training. A model relying on absolute positions must somehow generalize to these unseen combinations, either by interpolating from nearby positions or by pattern-matching to semantically similar content. Neither approach is principled. With relative encoding, every distance seen during training transfers to all positions at that distance, turning a difficult extrapolation problem into a simple reuse of already-learned representations.