Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Recurrent Memory › The Position Encoding Problem

Michael BrenndoerferUpdated July 6, 20253 min read

Language AI HandbookAll chapters

Part 18 of 67Long Context

Chapter 132Recurrent Memory › The Position Encoding Problem

Reading section 5 of 14 · The Position Encoding Problem · 3 minSection 5 of 14 · 3 min

Section 5 of 14 · 3 min read

Recurrent Memory › The Position Encoding ProblemOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Standard absolute position encodings break under segment-level recurrence. If we use learned or sinusoidal position embeddings based on absolute positions within each segment, we encounter a fundamental inconsistency.

Consider a token at position 5 in segment $\tau$ . In the previous segment $\tau - 1$ , there was also a token at position 5. Both receive the same absolute position encoding. But from the perspective of the current segment, these tokens are at very different distances: position 5 in the current segment is "here," while position 5 in the cached segment is 8 positions back (if segments have length 8).

The problem is even deeper than this. Absolute positions within a segment reset at the start of each segment. Position 0 in segment 1 and position 0 in segment 2 both have the same position encoding, but they are separated by the entire length of segment 1. When we concatenate their hidden states and ask the attention mechanism to use position information to understand their relationship, we give it the same encoding for tokens that are far apart. The attention mechanism cannot distinguish between "this is my first token" and "this is the first token in some previous segment."

Think of it like a book where every chapter starts its page numbers over from 1. If you're on page 5 of chapter 3 and want to cite page 5 of chapter 1, both references look identical in the page numbering system, but they point to completely different locations. The reader needs additional information (the chapter number) to disambiguate. Transformer-XL solves this analogous problem by replacing absolute position numbers with relative distance measures, so the position encoding always says "this token is $k$ steps before me" rather than "this token is at absolute position $n$ ."

In[14]:

Code

def demonstrate_position_conflict():
    """
    Show how absolute position encodings create confusion
    in segment-level recurrence.
    """
    segment_length = 8

    # Absolute positions assigned during each segment's processing
    segment_tau_minus_1_positions = list(
        range(segment_length)
    )  # [0, 1, 2, ..., 7]
    segment_tau_positions = list(range(segment_length))  # [0, 1, 2, ..., 7]

    # True temporal distances from perspective of segment tau
    # Cached tokens are at positions -8 to -1 relative to segment tau's start
    true_distances_cached = list(range(-segment_length, 0))  # [-8, -7, ..., -1]
    true_distances_current = list(range(segment_length))  # [0, 1, ..., 7]

    return {
        "cached_absolute_pos": segment_tau_minus_1_positions,
        "current_absolute_pos": segment_tau_positions,
        "cached_true_distance": true_distances_cached,
        "current_true_distance": true_distances_current,
    }


pos_info = demonstrate_position_conflict()

Out[15]:

Console

Position Encoding Conflict in Segment-Level Recurrence

Cached Segment (τ-1):
  Absolute positions (as encoded): [0, 1, 2, 3, 4, 5, 6, 7]
  True distance from current τ:    [-8, -7, -6, -5, -4, -3, -2, -1]

Current Segment (τ):
  Absolute positions (as encoded): [0, 1, 2, 3, 4, 5, 6, 7]
  True distance from current τ:    [0, 1, 2, 3, 4, 5, 6, 7]

Problem: Position 5 in cached segment and position 5 in current segment
both have the same absolute encoding, but their true distances differ by 8!

If absolute position encodings were used, the model would receive conflicting signals. Two tokens with identical position encodings would be at different temporal distances. The attention mechanism, which relies on position information to understand sequence structure, would be confused.

Transformer-XL solves this with relative position encodings. Instead of encoding absolute positions and adding them to token embeddings, the model directly encodes the relative distance between query and key positions in the attention computation itself.