RoBERTa › Fine-Tuning RoBERTa
Language AI HandbookAll chapters RoBERTa › Fine-Tuning RoBERTaOne section · Sections in this chapter13 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of RoBERTa's most common use cases is as a starting point for fine-tuning on downstream classification tasks. The fine-tuning process adds a small task-specific head on top of the pretrained encoder and trains the entire model on labeled task data. RoBERTa's stronger pretraining representations mean that it typically needs fewer labeled examples to reach a given performance level compared to BERT.
The standard fine-tuning procedure for classification tasks takes the CLS token representation from the final transformer layer, passes it through a linear classifier, and trains with cross-entropy loss. The pretrained weights provide a strong initialization, so training is usually stable and fast, typically converging in 3-10 epochs on standard benchmark tasks.
from transformers import RobertaForSequenceClassification, RobertaTokenizer
# Load RoBERTa with a classification head for binary sentiment classification
clf_model = RobertaForSequenceClassification.from_pretrained(
"roberta-base",
num_labels=2, # binary: positive / negative
hidden_dropout_prob=0.1,
attention_probs_dropout_prob=0.1,
)
tokenizer_clf = RobertaTokenizer.from_pretrained("roberta-base")
# Sample texts and labels (0 = negative, 1 = positive)
sample_texts = [
"This product exceeded all my expectations.",
"Absolutely terrible experience, would not recommend.",
"Decent quality, nothing special.",
"Outstanding service from start to finish.",
]
sample_labels = [1, 0, 0, 1]# Tokenize inputs
encoded = tokenizer_clf(
sample_texts,
padding=True,
truncation=True,
max_length=128,
return_tensors="pt",
)
labels_tensor = torch.tensor(sample_labels)
# Forward pass with labels to compute loss
clf_model.train()
outputs = clf_model(**encoded, labels=labels_tensor)Classification loss: 0.7359 Logits shape: torch.Size([4, 2]) Predicted classes: [0, 1, 0, 0] True classes: [1, 0, 0, 1]
The untrained classification head produces random initial predictions, as expected. After fine-tuning on a real labeled dataset, the model learns to map RoBERTa's rich contextual representations to task-relevant categories. For standard classification benchmarks, a few thousand labeled examples are usually sufficient to reach near-peak performance, thanks to the strong representational foundation from pretraining.
