Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

DeBERTa › Relative Position Encoding

Michael BrenndoerferUpdated July 20, 20259 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 176DeBERTa › Relative Position Encoding

Reading section 4 of 17 · Relative Position Encoding · 9 minSection 4 of 17 · 9 min

Section 4 of 17 · 9 min read

DeBERTa › Relative Position EncodingOne section · Sections in this chapter4 / 17 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The disentangled attention formula references position embeddings $P_{i|j}$ and $P_{j|i}$ , but we have not yet explained what these embeddings encode or how they are computed. This section develops the relative position encoding scheme that makes disentangled attention possible. Understanding relative positions matters for DeBERTa and for transformer architecture more broadly, as relative position encoding has become a widely-adopted design choice across many subsequent models.

The central challenge is turning "the distance between positions $i$ and $j$ " into a vector that can participate in dot product attention. The distance itself is just a signed integer: $j - i$ is positive when $j$ comes after $i$ and negative when $j$ comes before. We need a way to map this integer to a high-dimensional vector that the model can learn to use effectively. The solution also needs to handle arbitrarily long sequences without requiring an unbounded number of distinct embeddings.

From Absolute to Relative PositionsLink Copied

BERT uses absolute position embeddings: position 0 gets one learned vector, position 1 gets another, and so on. Each position in the sequence has a fixed identity tied to its location. While simple and effective for short, predictably-positioned sequences, this approach has a fundamental limitation: the model must learn separately that "position 2 attending to position 5" and "position 7 attending to position 10" both represent the same relationship of "3 tokens ahead." The model sees these as entirely different operations and must allocate separate representational capacity to each.

Relative position encoding captures the insight that what matters is the distance between tokens, not their absolute locations. Instead of encoding "I am at position 5," a token encodes "token $j$ is 3 positions ahead of me." This single representation applies whether the token pair sits at the start, middle, or end of the sequence. The model needs to learn this pattern once, and it transfers to every position in the sequence automatically.

Think of absolute positions as home addresses and relative positions as travel directions. Knowing that you are at "42 Main Street" requires knowing the entire street numbering system. Knowing that "the grocery store is 3 blocks north" works regardless of which street you start from. Relative encoding gives the model the directional language it needs to describe relationships without tying those descriptions to absolute coordinates.

The benefits of this approach are substantial:

Generalization across positions: Patterns learned at one location in the sequence automatically transfer to all other locations, reducing the amount of training data needed to learn positional syntax
Length flexibility: The model can handle sequences longer than those seen during training, since relative positions are defined in terms of distance rather than absolute index
Linguistic alignment: Grammar and syntax in most languages depend on relative word order. Subject-verb agreement, prepositional attachment, and modifier scope are all defined by relative proximity, not absolute position
Cleaner gradient flow: Because positional and content signals never add together into a single vector, gradients from positional tasks cannot interfere with learning in the content stream

The Relative Position FormulaLink Copied

Given query position $i$ and key position $j$ , the relative position embedding $P_{i|j}$ encodes the signed distance $i - j$ . Note the convention: DeBERTa uses $i - j$ rather than $j - i$ , so a positive value means the key is behind the query, and a negative value means the key is ahead of the query.

We face a practical constraint: we cannot have infinitely many embeddings for every possible distance. For a sequence of length $n$ , distances range from $-(n-1)$ to $+(n-1)$ , and for long sequences this would require thousands of embedding vectors. DeBERTa bounds relative positions to a maximum distance $k$ (typically 512), mapping all distances beyond $\pm k$ to the boundary embeddings. The mapping function converts a raw relative distance to an embedding table index:

\delta(i, j) = \begin{cases} 0 & \text{if } i - j \leq -k \\ 2k - 1 & \text{if } i - j \geq k \\ i - j + k & \text{otherwise} \end{cases}

where:

$\delta(i, j)$ : the embedding table index we use to look up the position embedding for the $i$ -to- $j$ relationship
$i - j$ : the raw signed relative distance between positions $i$ (query) and $j$ (key)
$k$ : the maximum relative distance encoded with a unique embedding; all distances beyond $\pm k$ are clipped

Why does this formula make sense? Notice that the three cases implement a "clip-and-shift" strategy. When the key is more than $k$ positions ahead ( $i - j \leq -k$ ), we cannot distinguish exactly how far ahead it is, so all such positions share the same index 0. Similarly, when the key is more than $k$ positions behind ( $i - j \geq k$ ), all such positions share the maximum index $2k - 1$ . For the common case where the distance falls within $(-k, k)$ , we shift by $+k$ to convert negative distances to non-negative indices. A distance of $-k$ maps to index 0, distance 0 maps to index $k$ , and distance $k-1$ maps to index $2k-1$ .

Why 2k Embeddings?Link Copied

The total number of embeddings is $2k$ because we need to represent distances from $-(k)$ to $+(k-1)$ , a range of $2k$ distinct values:

Negative distances from $-k$ to $-1$ : these are $k$ distinct values representing the key being ahead of the query
Zero and positive distances from $0$ to $k-1$ : these are $k$ distinct values representing the key being at the same position or behind the query
Total: exactly $2k$ embeddings required

For the default $k = 512$ , this gives 1024 position embeddings, which is quite manageable memory-wise while covering the vast majority of practical attention distances. Sequences shorter than 512 tokens (the most common case in NLU fine-tuning) use all embeddings at full resolution. Only token pairs further than 512 positions apart share boundary embeddings, and such long-range dependencies tend to carry less syntactic signal anyway.

In[8]:

Code

class RelativePositionEmbedding(nn.Module):
    """Relative position embeddings for DeBERTa."""

    def __init__(
        self, max_relative_positions: int = 512, hidden_size: int = 768
    ):
        super().__init__()
        self.max_relative_positions = max_relative_positions
        # 2k embeddings: positions from -k to k-1
        self.embeddings = nn.Embedding(2 * max_relative_positions, hidden_size)

    def forward(self, seq_len: int) -> torch.Tensor:
        """
        Generate relative position embeddings for a sequence.

        Args:
            seq_len: Length of the sequence

        Returns:
            Relative position embedding matrix of shape (seq_len, seq_len, hidden_size)
        """
        # Create position indices
        positions = torch.arange(seq_len)

        # Compute relative positions: positions[i] - positions[j]
        relative_positions = positions.unsqueeze(1) - positions.unsqueeze(0)

        # Clip to valid range and shift to positive indices
        k = self.max_relative_positions
        relative_positions = torch.clamp(relative_positions, -k, k - 1)
        relative_positions = relative_positions + k  # Shift to [0, 2k)

        # Look up embeddings
        return self.embeddings(relative_positions)

Out[9]:

Console

Relative Position Embedding:
  Sequence length: 6
  Output shape: torch.Size([6, 6, 64])
  Shape interpretation: (query_pos, key_pos, hidden_dim)

Relative position matrix (before clipping and shifting):
[[ 0 -1 -2 -3 -4 -5]
 [ 1  0 -1 -2 -3 -4]
 [ 2  1  0 -1 -2 -3]
 [ 3  2  1  0 -1 -2]
 [ 4  3  2  1  0 -1]
 [ 5  4  3  2  1  0]]

The output shape (6, 6, 64) provides a unique embedding for each query-key position pair in the sequence. For a sequence of length $n$ , this creates an $n \times n$ matrix of position embeddings, one for each possible (query, key) combination. This is the key data structure that feeds into the content-to-position and position-to-content attention terms. The relative position matrix shows the raw signed distances before clipping and shifting: notice the antisymmetry, where entry $(i, j)$ is the negative of entry $(j, i)$ . This is why $P_{i|j}$ and $P_{j|i}$ are different embeddings for the same pair of positions: the same relationship looks different depending on which token is asking the question.

Out[10]:

Visualization

Heatmap showing cosine similarity between relative position embeddings with a diagonal band structure. — Cosine similarity between relative position embeddings for positions ranging from -16 to +15. Sinusoidal embeddings produce higher similarity between nearby relative positions and lower similarity between distant ones, creating the band structure visible along the diagonal. This pattern means the model treats similar positional distances similarly, supporting generalization across sequence positions.

The similarity matrix reveals that nearby relative positions (for example, +2 and +3) have more similar embeddings than distant ones (for example, +2 and +15). This structure emerges naturally from sinusoidal initialization and persists through training because similar distances should carry similar information. It means the model can smoothly interpolate between distances it has seen: if it learns that "2 positions ahead" signals something important, it will partially apply that knowledge to "3 positions ahead" as well.

Out[11]:

Visualization

Heatmap showing relative position indices from -5 to +5 for a 6-token sequence with numerical annotations. — Relative position indices for a 6-token sequence showing the signed distance from each query position (row) to each key position (column). Entry (i, j) gives the value i minus j: negative entries (blue) indicate the key is ahead of the query, and positive entries (red) indicate the key is behind. The antisymmetric pattern reflects the asymmetry between the content-to-position and position-to-content attention terms.

The relative position matrix shows a clear antisymmetric structure: position 0 sees position 3 as "-3" (three positions ahead), while position 3 sees position 0 as "+3" (three positions behind). This asymmetry is captured in the embeddings and used differently in the content-to-position versus position-to-content attention terms, which is why both $P_{i|j}$ and $P_{j|i}$ appear in the formula rather than just one of them.