Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Relative Position Encoding › Clipping Relative Positions

Michael BrenndoerferUpdated June 2, 20255 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 99Relative Position Encoding › Clipping Relative Positions

Reading section 5 of 16 · Clipping Relative Positions · 5 minSection 5 of 16 · 5 min

Section 5 of 16 · 5 min read

Relative Position Encoding › Clipping Relative PositionsOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formulation above assumes we have a separate embedding for every possible relative position. But this creates a practical problem: for a sequence of length $n$ , relative positions range from $-(n-1)$ to $(n-1)$ , requiring $2n-1$ embeddings. For a sequence of length 512, that's 1023 different embeddings per dimension, for both keys and values.

This approach has two flaws. First, memory grows linearly with sequence length. Second, distant positions appear infrequently during training. In a 128-token sequence, only one position pair has offset $+127$ (position 0 to position 127), while thousands of pairs have offset $+1$ . The embedding for offset $+127$ would be severely undertrained: the gradient updates for that embedding come from only one pair per sequence, while the embedding for offset $+1$ receives gradient contributions from 127 pairs. The result is that distant-position embeddings never become reliable.

Shaw et al. address both problems with a simple insight: fine-grained distance matters more locally than globally. The difference between "3 positions back" and "4 positions back" may be linguistically significant, as it could distinguish an adjective-noun relationship from a determiner-noun relationship. But the difference between "100 positions back" and "101 positions back" rarely carries meaning. Both are simply "far away." The model can usefully represent "very distant" as a single category without losing important information.

This motivates clipping: relative positions beyond a maximum distance $k$ are clamped to $\pm k$ :

\text{clip}(j - i, -k, k)

where:

$j - i$ : the raw relative position (offset from position $i$ to position $j$ )
$k$ : the maximum relative position to distinguish (a hyperparameter)
$\text{clip}(x, a, b)$ : clamps $x$ to the range $[a, b]$ , returning $a$ if $x < a$ , $b$ if $x > b$ , and $x$ otherwise

With clipping, we need only $2k + 1$ relative position embeddings regardless of sequence length. Position pairs further than $k$ apart share the same "far away" embedding, either $\mathbf{a}^K_{-k}$ for "far backward" or $\mathbf{a}^K_{+k}$ for "far forward."

The choice of $k$ reflects a trade-off between expressiveness and learnability. Typical values range from 16 to 128, depending on the expected locality of relevant patterns in the data. For sentence-level tasks where the longest dependencies span 10-20 tokens, $k = 16$ often suffices. For document-level tasks with paragraph-scale dependencies, $k = 64$ or $k = 128$ gives the model more distance resolution. In practice, the performance difference between $k = 32$ and $k = 64$ is often small, but the difference between $k = 4$ and $k = 16$ can be significant for tasks involving long-range grammatical dependencies.

In[6]:

Code

def clip_relative_position(j, i, max_dist):
    """
    Compute clipped relative position.

    Args:
        j: Key/value position
        i: Query position
        max_dist: Maximum relative distance to distinguish

    Returns:
        Clipped relative position in range [-max_dist, max_dist]
    """
    rel_pos = j - i
    return max(-max_dist, min(max_dist, rel_pos))


# Demonstrate clipping
max_k = 4
positions = list(range(10))
query_pos = 5

Out[7]:

Console

Query position: 5
Maximum relative distance (k): 4

Position | Raw Offset | Clipped Offset
------------------------------------------
    0    |     -5     |      -4
    1    |     -4     |      -4
    2    |     -3     |      -3
    3    |     -2     |      -2
    4    |     -1     |      -1
    5    |     +0     |      +0
    6    |     +1     |      +1
    7    |     +2     |      +2
    8    |     +3     |      +3
    9    |     +4     |      +4

The table reveals how clipping works in practice. Positions 0 and 1 have raw offsets of -5 and -4, but both clip to -4 because they exceed our maximum distance of 4. Position 9 has raw offset +4, which equals the maximum so no clipping occurs. All positions beyond the clipping threshold share the same "far away" embedding, which captures the intuition that distinguishing between "very far" and "extremely far" rarely matters for language understanding.

Out[8]:

Visualization

Heatmap showing clipped relative positions for a 10x10 matrix with k=4, displaying values from -4 to +4. — Effect of clipping on the relative position matrix for a 10-token sequence with k=4. Each cell shows the clipped relative offset used to index the embedding table. The diagonal band of distinct values (from -4 to +4) widens as k increases, while the saturated corners (darkest blue and darkest red) show positions that share the same embedding because they are further than k steps apart.

Notice the diagonal band structure: each anti-diagonal contains the same relative position. The corners, where positions are far apart, are clipped to the boundary values $\pm k$ . This Toeplitz-like structure is key to efficient implementation because it means position biases repeat in a predictable pattern that can be computed once and reused.