Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Relative Position Encoding › Comparison with Absolute Position Encoding

Michael BrenndoerferUpdated June 2, 20252 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 99Relative Position Encoding › Comparison with Absolute Position Encoding

Reading section 10 of 16 · Comparison with Absolute Position Encoding · 2 minSection 10 of 16 · 2 min

Section 10 of 16 · 2 min read

Relative Position Encoding › Comparison with Absolute Position EncodingOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's directly compare relative and absolute position encodings on a simple task: detecting whether a token is attending to its immediate neighbors versus distant positions.

In[22]:

Code

def analyze_position_bias(attention_weights, max_offset=5):
    """
    Analyze average attention by relative offset.
    """
    seq_len = attention_weights.shape[0]
    offset_weights = {}

    for offset in range(-max_offset, max_offset + 1):
        weights_at_offset = []
        for i in range(seq_len):
            j = i + offset
            if 0 <= j < seq_len:
                weights_at_offset.append(attention_weights[i, j])
        if weights_at_offset:
            offset_weights[offset] = np.mean(weights_at_offset)

    return offset_weights


# Analyze position patterns in our relative attention
offset_analysis = analyze_position_bias(combined_weights)

Out[23]:

Visualization

Bar chart showing average attention weight for relative offsets from -5 to +5, with the self-attention bar at offset 0 highlighted in red below the uniform baseline. — Average attention weight by relative offset for the test sequence. The red bar at offset 0 marks self-attention. The dashed line marks the uniform baseline (equal weight to all positions). Bars above the baseline indicate offsets that receive more attention than chance, while bars below indicate less. In trained models, this profile takes on a characteristic shape that reflects the model's learned distance preferences for a given task.

The profile shows how attention is distributed by relative distance. In this random initialization, patterns are noisy, but trained models develop clear preferences: immediate neighbors typically receive more attention, with the pattern decaying for distant positions. The key observation is that this profile is the same at every absolute position in the sequence. A token at position 50 and a token at position 500 both use the same distance-indexed biases. Absolute position encoding cannot achieve this: the bias for "position 50 attending to position 52" has no relationship to the bias for "position 500 attending to position 502," because the absolute embedding at position 52 is entirely independent from the absolute embedding at position 502.