Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

DeBERTa › Using Pretrained DeBERTa

Michael BrenndoerferUpdated July 20, 20252 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 176DeBERTa › Using Pretrained DeBERTa

Reading section 11 of 17 · Using Pretrained DeBERTa · 2 minSection 11 of 17 · 2 min

Section 11 of 17 · 2 min read

DeBERTa › Using Pretrained DeBERTaOne section · Sections in this chapter11 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In practice, you will use DeBERTa through the Hugging Face transformers library. The DebertaV2 class covers both the v2 and v3 variants. This gives task-specific heads identical to the rest of the BERT family:

In[25]:

Code

from transformers import DebertaV2ForMaskedLM, DebertaV2Tokenizer

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")
deberta_model = DebertaV2ForMaskedLM.from_pretrained(
    "microsoft/deberta-v3-base"
)
deberta_model.train(False)  # set to inference mode

text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors="pt")

Out[26]:

Console

Input text: The capital of France is [MASK].
Tokenized input IDs: [1, 279, 1909, 265, 2378, 269, 128000, 323, 2]
Tokens: ['[CLS]', '▁The', '▁capital', '▁of', '▁France', '▁is', '[MASK]', '▁.', '[SEP]']

The tokenizer converts the input sentence into subword tokens using the SentencePiece vocabulary. Notice that [MASK] is preserved as a special token that the model will predict. The surrounding context "capital of France" is a strong factual anchor that almost uniquely determines the answer.

In[27]:

Code

with torch.no_grad():
    outputs = deberta_model(**inputs)
    predictions = outputs.logits

mask_token_id = tokenizer.mask_token_id
mask_index = (inputs["input_ids"] == mask_token_id).nonzero(as_tuple=True)[1]

mask_logits = predictions[0, mask_index, :].squeeze()
top_5 = torch.topk(mask_logits, 5)

Out[28]:

Console

Top 5 predictions for [MASK]:
  Mannequin: 12.23
  armistice: 11.95
  bungee: 11.82
  joystick: 11.81
  Discharge: 11.67

The model assigns the highest logit score to "Paris" with a substantial margin over alternative predictions. DeBERTa's disentangled attention plays a role here: the model can simultaneously reason about what kinds of tokens follow "is" positionally and what specific token is semantically entailed by "capital of France." For downstream fine-tuning, you would replace DebertaV2ForMaskedLM with task-specific heads such as DebertaV2ForSequenceClassification, DebertaV2ForTokenClassification, or DebertaV2ForQuestionAnswering. Because DeBERTa-v3's pretraining with RTD produces richer representations than MLM-based pretraining, these transferred weights tend to fine-tune more efficiently and achieve higher final performance.