DeBERTa › Worked Example: Tracing the Attention Computation

To make the mathematics concrete, let's trace a minimal numerical example through the full disentangled attention computation. We will use a two-token sequence with a two-dimensional hidden space and a single attention head, keeping all numbers small enough to follow by hand.

Suppose we have tokens $A$ and $B$ with content vectors:

$$H_A = [1, 0], \quad H_B = [0, 1]$$

And suppose our (simplified, square) projection matrices are all the identity: $W_q^c = W_k^c = W_q^p = W_k^p = I$. For relative positions, suppose:

$$P_{A|B} = [0.5, 0.5] \quad \text{(B is 1 position ahead of A)}$$ $$P_{B|A} = [-0.5, 0.5] \quad \text{(A is 1 position behind B)}$$

Step 1: Compute the three attention components for the A-to-B score $A_{AB}$.

Content-to-content:

$$\begin{aligned} H_A W_q^c W_k^{c\top} H_B^\top &= [1, 0] \cdot I \cdot I \cdot [0, 1]^\top \\ &= [1, 0] \cdot [0, 1]^\top \\ &= 0 \end{aligned}$$

The two tokens have orthogonal content vectors, so the semantic similarity is zero.

Content-to-position:

$$\begin{aligned} H_A W_q^c W_k^{p\top} P_{A|B}^\top &= [1, 0] \cdot I \cdot I \cdot [0.5, 0.5]^\top \\ &= [1, 0] \cdot [0.5, 0.5]^\top \\ &= 0.5 \end{aligned}$$

Token $A$'s content (dimension 0 active) aligns with the first component of the position embedding for "1 ahead."

Position-to-content:

$$\begin{aligned} P_{B|A} W_q^p W_k^{c\top} H_B^\top &= [-0.5, 0.5] \cdot I \cdot I \cdot [0, 1]^\top \\ &= [-0.5, 0.5] \cdot [0, 1]^\top \\ &= 0.5 \end{aligned}$$

Token $B$'s relative position from $A$'s perspective aligns with token $B$'s content vector.

Step 2: Sum the three components.

$$A_{AB} = 0 + 0.5 + 0.5 = 1.0$$

The key insight from this example: even though the semantic content of $A$ and $B$ is completely orthogonal (zero content-to-content score), the model still produces a non-zero attention score because the positional components contribute. This is exactly the expressive advantage of disentangled attention: the model can attend based on positional relationship even when semantic similarity is low. Standard BERT attention would compute $[1,0] \cdot [0,1]^\top = 0$ and conclude that $A$ should not attend to $B$ at all, even if their relative position is syntactically important.

Step 3: Scale and normalize.

With a single head and $d = 2$, the scaling factor is $1/\sqrt{2} \approx 0.707$. The scaled score is $1.0 \times 0.707 = 0.707$. For a two-token sequence we compute both $A_{AB}$ and $A_{AA}$, apply softmax across both, and use the resulting probabilities to weight the value vectors. The full softmax step would involve computing $A_{AA}$ by the same three-component method, but the key point is clear: the positional components changed what attention weights the model will learn.