DeBERTa › DeBERTa Improvements: Two-Stream Attention
Language AI HandbookAll chapters DeBERTa › DeBERTa Improvements: Two-Stream AttentionOne section · Sections in this chapter9 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
DeBERTa also incorporates ideas from XLNet. Specifically, it uses a variant of XLNet's two-stream attention during pretraining, which separates content and query representations for masked token prediction. This addition addresses a subtle information leakage problem in the standard MLM setup.
When predicting a masked token, the model should not see the token's own content (that would trivially reveal the answer), but it should know the token's position (to understand where in the sentence the prediction occurs). Two-stream attention achieves this separation cleanly. The two streams operate in parallel:
- Content stream: Sees all tokens including the current position's content. This is the standard self-attention path, processing all available information.
- Query stream: Sees all tokens except the current position's content. The current position contributes only its positional embedding to the query, not its semantic content.
During masked language modeling, predictions are made using the query stream representations. Because the query stream cannot see the current token's content, the model is forced to infer the masked value from context alone. After pretraining, the query stream is discarded and only the content stream is used for downstream tasks, so there is no inference overhead.
