DeBERTa › Summary

DeBERTa introduced several architectural innovations that systematically addressed limitations in BERT's attention mechanism, and each innovation has a clear motivation rooted in what standard attention cannot express.

Disentangled attention separates content and position into distinct representation streams, then computes three attention components: content-to-content, content-to-position, and position-to-content. This gives the model explicit control over how semantic meaning and structural location interact during attention, rather than implicitly combining them through a single dot product. The result is an attention score that can capture semantic similarity, positional syntax, and their interaction as independent, explicitly-parameterized quantities.

Relative position encoding replaces BERT's absolute positions with signed distances between token pairs. This generalizes better across sequence lengths, transfers positional patterns from one location to all locations in the sequence, and aligns with the linguistic reality that grammar depends on relative word order rather than absolute position. The clipping mechanism bounds the embedding table to $2k$ entries while preserving fine-grained resolution within a 512-token window.

Enhanced Mask Decoder delays absolute position injection until just before MLM prediction. The encoder uses only relative positions throughout its stack, keeping positional signals clean and separate. The EMD then adds absolute position information in two dedicated layers at the end, giving the prediction head access to both the rich relative structural representations from the encoder and the absolute position signal that identifies where in the sequence the prediction occurs.

DeBERTa-v2 added n-gram embeddings via depthwise convolution, expanded vocabulary size to 128K tokens, and demonstrated that the architecture scales well to 1.5 billion parameters. The nGiE module provides a lightweight inductive bias that primes token representations with local neighborhood structure before the transformer layers process them.

DeBERTa-v3 adopted ELECTRA-style replaced token detection, training the main model as a discriminator that identifies generator-created token substitutions. This objective trains on all token positions rather than just 15%. This provides 6.7x more gradient signal per batch and producing richer representations that consistently outperform MLM-trained models of comparable size.

The combination of these techniques produced a model that consistently outperforms BERT and RoBERTa, along with ALBERT, on challenging NLU benchmarks. For applications where accuracy justifies additional computation, DeBERTa-v3 represents the current practical state of the art in encoder-only transformers, and the architectural principles it introduced, particularly the decomposition of attention into explicit semantic and positional components, continue to influence how researchers think about building better transformer architectures.