Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

RoBERTa › Using Pre-trained RoBERTa

Michael BrenndoerferUpdated July 18, 20252 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 173RoBERTa › Using Pre-trained RoBERTa

Reading section 10 of 17 · Using Pre-trained RoBERTa · 2 minSection 10 of 17 · 2 min

Section 10 of 17 · 2 min read

RoBERTa › Using Pre-trained RoBERTaOne section · Sections in this chapter10 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In practice, you'll likely use RoBERTa through the Hugging Face transformers library rather than training from scratch. This section covers the most common usage patterns: masked language model inference, representation extraction, and fine-tuning preparation. Understanding how to interact with the model is essential before you can apply it to real tasks.

RoBERTa uses a byte-pair encoding tokenizer with a vocabulary of 50,265 tokens, slightly larger than BERT's WordPiece vocabulary of about 30,000. The tokenizer was trained on the same large corpus used for pretraining, which means it is well-adapted to the full range of text genres that RoBERTa was trained on. Unlike BERT, which uses [CLS] and [SEP] as special tokens, RoBERTa follows GPT-2's conventions and uses <s> and </s>. This is a naming difference only; the functional roles are the same.

Load and use the tokenizer and model as follows:

In[23]:

Code

from transformers import RobertaForMaskedLM, RobertaTokenizer

# Load pre-trained RoBERTa
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaForMaskedLM.from_pretrained("roberta-base")
model.eval()

# Prepare input with a masked token
text = "The capital of France is <mask>."
inputs = tokenizer(text, return_tensors="pt")

Out[24]:

Console

Input text: The capital of France is <mask>.
Tokenized input IDs: [0, 133, 812, 9, 1470, 16, 50264, 4, 2]
Tokens: ['<s>', 'The', 'Ġcapital', 'Ġof', 'ĠFrance', 'Ġis', '<mask>', '.', '</s>']

In[25]:

Code

# Get predictions for the masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits

# Find the position of the mask token
mask_token_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(
    as_tuple=True
)[1]

# Get top 5 predictions
mask_logits = predictions[0, mask_token_index, :].squeeze()
top_5 = torch.topk(mask_logits, 5)

Out[26]:

Console


Top 5 predictions for <mask>:
   Paris: 21.54
   Lyon: 19.12
   Nice: 16.30
   Nancy: 15.47
   Napoleon: 14.85

RoBERTa correctly predicts "Paris" with high confidence. The model has learned rich representations of factual knowledge through its MLM pretraining.

The MLM interface is useful for probing what the model knows. You can test factual knowledge ("The capital of France is <mask>."), grammatical constraints ("The children is/are playing outside."), or semantic preferences ("She poured the hot coffee into her <mask>."). These probes reveal aspects of what the model has internalized during pretraining and can diagnose potential biases or gaps in its knowledge.