DeBERTa › Disentangled Attention Formulation

Now that we understand why entangled representations limit the model's expressiveness, let's develop the mathematical framework for disentangled attention. The key insight is deceptively simple: instead of computing one attention score that mixes content and position, we compute separate scores for each type of relationship and combine them. This requires maintaining two separate representation streams and three families of projection matrices rather than one, but the payoff is an attention score that is explicitly interpretable in terms of what it is measuring.

Before diving into the formula, it helps to think clearly about what information each stream carries. The content stream $H_i$ encodes what token $i$ means: its semantic role, syntactic category, and all the contextual information accumulated from previous transformer layers. The position stream $P_{i|j}$ encodes where token $j$ is relative to token $i$: it captures the distance and direction between two positions without saying anything about what those positions contain. Keeping these streams separate means that when the model updates them during training, gradients for semantic learning do not corrupt positional encodings and vice versa.

When token $i$ decides how much to attend to token $j$, it implicitly asks several questions. Think of each attention component as the model's way of answering one specific question:

1. "Is the meaning at position $j$ relevant to my meaning?" This is a pure semantic question. The word "bank" should attend strongly to "river" or "money" based on meaning alone, regardless of where these words appear in the sentence. The content-to-content component answers this.

2. "Given what I mean, is the relative position of $j$ important?" Certain words care about specific positional relationships. A verb might strongly attend to whatever appears one or two positions before it (likely the subject), regardless of what word occupies that position. The content-to-position component answers this, letting the model's semantic content drive a positionally-aware query.

3. "Given my position relative to $j$, is the content at $j$ important?" Position can make content more or less relevant. The position-to-content component allows the model to learn that certain relative positions are expected to carry certain kinds of content, for example that the token directly following a determiner is typically a noun or adjective.

Standard attention conflates all three questions into a single dot product. The model is forced to find weight matrices that simultaneously capture all three relationships through a single computation. Disentangled attention answers each explicitly, giving the model more capacity to specialize each component for what it does best.

With this intuition, we can write the full attention score as a sum of three terms:

$$A_{ij} = \underbrace{H_i W_q^c W_k^{c\top} H_j^\top}_{\text{content-to-content}} + \underbrace{H_i W_q^c W_k^{p\top} P_{i|j}^\top}_{\text{content-to-position}} + \underbrace{P_{j|i} W_q^p W_k^{c\top} H_j^\top}_{\text{position-to-content}}$$

where:

• $H_i$, $H_j$: Content vectors at positions $i$ and $j$, encoding what the tokens mean with no positional information mixed in
• $P_{i|j}$: The relative position embedding from $i$'s perspective looking toward $j$. If $j$ is 3 positions ahead, this encodes the concept "+3 ahead"
• $P_{j|i}$: The relative position embedding from $j$'s perspective looking toward $i$. For the same pair, this encodes the concept "-3 behind"
• $W_q^c$, $W_k^c$: Learned projection matrices that map content vectors into query and key spaces for semantic comparison
• $W_q^p$, $W_k^p$: Learned projection matrices that map position embeddings into query and key spaces for positional comparison

The superscripts $c$ (content) and $p$ (position) distinguish which representation type each matrix operates on. Notice that content and position projections are entirely separate: there are no shared weights between the two streams. This makes the disentanglement structural rather than superficial.

Why does the formula make sense? Notice that each term is a bilinear form: a left vector times a matrix product times a right vector. The matrices $W_q$ and $W_k$ are the standard query and key projections from attention, but now specialized to operate on either content or position. The dot product structure is preserved, which means the entire mechanism slots naturally into standard scaled dot-product attention: we compute these three terms, sum them, scale by $1/\sqrt{d}$, apply softmax, and use the resulting weights to aggregate value vectors.

Let's trace through what each component computes in detail.

Term 1: Content-to-Content

$$H_i W_q^c W_k^{c\top} H_j^\top$$

This measures semantic similarity between tokens $i$ and $j$ without any positional influence. The content at position $i$ is projected into a query space ($H_i W_q^c$), and the content at position $j$ is projected into a key space ($H_j W_k^c$). Their dot product reveals how semantically aligned the two tokens are. This is functionally equivalent to standard BERT attention, except that $H_i$ and $H_j$ here carry only content information, not the blended content-plus-position signal that BERT uses.

Term 2: Content-to-Position

$$H_i W_q^c W_k^{p\top} P_{i|j}^\top$$

Here, the content at $i$ attends to the relative position of $j$. Notice that $H_i$ is projected with the content query matrix ($W_q^c$), while the position embedding $P_{i|j}$ is projected with the position key matrix ($W_k^p$). The cross-stream projection is intentional: we want the model to learn patterns like "verbs (semantic content) attend strongly to relative position -2 (where subjects tend to appear)." This term allows semantic identity to drive positional queries.

Term 3: Position-to-Content

$$P_{j|i} W_q^p W_k^{c\top} H_j^\top$$

This flips the relationship: the relative position from $j$'s perspective attends to the content at $j$. The position embedding becomes the query and content becomes the key. This enables patterns like "the slot two positions before me is expected to contain a noun." The position-to-content term lets structural position drive content-sensitive queries, capturing word-order regularities that are orthogonal to semantic similarity.

You might wonder: why not include a fourth term, $P_{i|j} W_q^p W_k^{p\top} P_{j|i}^\top$, for position-to-position attention?

The DeBERTa authors experimented with this fourth component and found it provides negligible benefit. Intuitively, relative positions are already informative on their own. Knowing that two positions are 3 apart does not become more useful by also considering that "3 apart" attends to "3 apart." The position-to-position signal is essentially constant for any pair at the same distance, contributing a fixed additive offset rather than any token-specific variation. That offset is redundant with the scaling factor applied before softmax. The computation is not worthwhile, so the authors omit it.

To build intuition for how the three components interact, let's examine their typical magnitudes and how they combine: