Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

DeBERTa › The Problem with Entangled Attention

Michael BrenndoerferUpdated July 20, 20255 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 176DeBERTa › The Problem with Entangled Attention

Reading section 2 of 17 · The Problem with Entangled Attention · 5 minSection 2 of 17 · 5 min

Section 2 of 17 · 5 min read

DeBERTa › The Problem with Entangled AttentionOne section · Sections in this chapter2 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before we can appreciate what DeBERTa fixes, we need to understand precisely what is wrong with BERT's original approach. The limitation is subtle: nothing in BERT is technically broken. But by merging content and position into a single vector early in the pipeline, the model forecloses certain kinds of reasoning that would otherwise be natural.

Standard BERT attention combines content and position by adding position embeddings to token embeddings at the input layer. Every downstream computation then operates on this mixed signal. The initial hidden representation for a sequence is:

H_0 = E_{\text{token}} + E_{\text{position}}

where:

$H_0$ : the initial hidden representation before any transformer layers, carrying both semantic and positional information from the very first step
$E_{\text{token}}$ : the token embedding looked up from the vocabulary embedding table, encoding what the word means in isolation
$E_{\text{position}}$ : the position embedding encoding the token's absolute position in the sequence, learned during pretraining

From this point forward, every hidden state is a mixture of content and position information. When computing attention scores, the query and key vectors both contain this entangled representation:

A_{ij} = \frac{Q_i \cdot K_j}{\sqrt{d}}

where:

$A_{ij}$ : the attention score between query position $i$ and key position $j$ , a single number representing how much token $i$ should attend to token $j$
$Q_i$ : the query vector at position $i$ , computed by projecting the entangled hidden state $H_i$ through a learned weight matrix $W_Q$
$K_j$ : the key vector at position $j$ , computed by projecting the entangled hidden state $H_j$ through a learned weight matrix $W_K$
$d$ : the dimension of the query and key vectors, used in the denominator to prevent dot products from growing large as $d$ increases
$Q_i \cdot K_j$ : the dot product measuring alignment between query and key vectors

Since $Q_i$ and $K_j$ each encode both the content at positions $i$ and $j$ and the absolute positions themselves, the model cannot separately ask "what content is at position $j$ ?" and "what is the relative position of $j$ to $i$ ?" These questions are conflated in a single dot product.

Entangled vs Disentangled Attention

In entangled attention (BERT), content and position are combined before attention, so the model cannot reason about them independently. In disentangled attention (DeBERTa), content and position maintain separate representations, allowing the model to compute distinct attention scores for content-content relationships and position-position relationships. The final score is a principled sum of explicit components rather than a single implicit mixture.

Consider the sentence "The cat sat on the mat." When determining how "sat" should attend to "cat," two distinct factors are at play. The first is a content relationship: "sat" is a verb that frequently takes animate subjects, and "cat" is exactly the kind of noun that appears as a subject. The second is a positional relationship: "cat" appears two positions before "sat," which is a typical subject-verb distance in English. In BERT, these two signals get folded together before attention is computed. The model learns to work with the mixture, but it cannot apply each signal independently to build a richer picture of the relationship.

DeBERTa computes them separately and combines them, giving each factor explicit representation in the attention computation. The result is a richer attention score that can simultaneously capture "these tokens are semantically related" and "the relative distance between them is consistent with a subject-verb dependency."

There is also a generalization argument. When BERT learns that "the word at absolute position 4" attends to "the word at absolute position 7," that pattern is tied to those specific absolute positions. A similar subject-verb relationship at positions 12 and 15 looks completely different to the model. With relative encoding, the model learns that "subject at distance -2" matters and can apply that pattern anywhere in the sequence. This is why relative position representations generalize better, particularly to sequences longer than those seen in training.

In[4]:

Code

def demonstrate_entanglement_problem():
    """Show how BERT entangles content and position."""
    # Simulated embeddings
    vocab_size, hidden_dim, seq_len = 30000, 64, 8

    # Token and position embeddings
    token_emb = nn.Embedding(vocab_size, hidden_dim)
    pos_emb = nn.Embedding(seq_len, hidden_dim)

    # Sample input (typical BERT token IDs for "[CLS] the cat sat on mat [SEP] [PAD]")
    token_ids = torch.tensor([[101, 1996, 4937, 2006, 2026, 13523, 102, 0]])
    positions = torch.arange(seq_len).unsqueeze(0)

    # BERT-style: add embeddings (entangled)
    bert_hidden = token_emb(token_ids) + pos_emb(positions)

    # DeBERTa-style: keep separate (disentangled)
    content = token_emb(token_ids)
    position = pos_emb(positions)

    return bert_hidden, content, position

Out[5]:

Console

BERT-style entangled representation:
  Shape: torch.Size([1, 8, 64])
  Contains: content + position (inseparable)

DeBERTa-style disentangled representations:
  Content shape: torch.Size([1, 8, 64])
  Position shape: torch.Size([1, 8, 64])
  Contains: separate streams that can interact explicitly

Both approaches produce representations of the same shape on the surface, but the DeBERTa style maintains two separate tensors throughout. This separation allows the attention mechanism to explicitly model how content relates to content versus how content relates to position, rather than conflating these two signals in every downstream operation. The shapes look identical, but the information flows very differently through the network.