Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

DeBERTa › Performance Comparison

Michael BrenndoerferUpdated July 20, 20252 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 176DeBERTa › Performance Comparison

Reading section 12 of 17 · Performance Comparison · 2 minSection 12 of 17 · 2 min

Section 12 of 17 · 2 min read

DeBERTa › Performance ComparisonOne section · Sections in this chapter12 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

DeBERTa achieved state-of-the-art results on numerous benchmarks when released. The progression from the original model to v3 shows steady, consistent improvements:

In[29]:

Code

# Benchmark results from papers (approximate)
benchmark_results = {
    "BERT-Large": {
        "params": 340,
        "mnli": 86.7,
        "qnli": 92.7,
        "sst2": 94.9,
        "squad_v2": 81.9,
    },
    "RoBERTa-Large": {
        "params": 355,
        "mnli": 90.2,
        "qnli": 94.7,
        "sst2": 96.4,
        "squad_v2": 89.4,
    },
    "ALBERT-xxLarge": {
        "params": 235,
        "mnli": 90.8,
        "qnli": 95.3,
        "sst2": 96.9,
        "squad_v2": 90.2,
    },
    "DeBERTa-Large": {
        "params": 350,
        "mnli": 91.1,
        "qnli": 95.8,
        "sst2": 96.8,
        "squad_v2": 90.7,
    },
    "DeBERTa-v3-Large": {
        "params": 304,
        "mnli": 91.8,
        "qnli": 96.0,
        "sst2": 97.2,
        "squad_v2": 91.4,
    },
}

Out[30]:

Console

Model Performance Comparison:
----------------------------------------------------------------------
Model                Params     MNLI       QNLI       SST-2      SQuAD 2   
----------------------------------------------------------------------
BERT-Large              340M     86.7      92.7      94.9      81.9
RoBERTa-Large           355M     90.2      94.7      96.4      89.4
ALBERT-xxLarge          235M     90.8      95.3      96.9      90.2
DeBERTa-Large           350M     91.1      95.8      96.8      90.7
DeBERTa-v3-Large        304M     91.8      96.0      97.2      91.4

The progression from BERT to DeBERTa-v3 shows steady improvements across all benchmarks. MNLI accuracy improves by over 5 points from BERT-Large to DeBERTa-v3-Large, while SQuAD v2 gains nearly 10 points. importantly, DeBERTa-v3-Large achieves these results with fewer parameters than RoBERTa-Large (304M vs 355M). This shows that architectural innovations can outperform simple parameter scaling.

Out[31]:

Visualization

Grouped bar chart comparing MNLI, QNLI, SST-2, and SQuAD v2 scores across five model variants. — Performance across four NLU benchmarks for five BERT-family models. DeBERTa-v3 achieves the highest scores on all four benchmarks despite having fewer parameters than RoBERTa-Large. This shows that architectural and pretraining improvements can outperform simple parameter scaling.