BERT Fine-tuning › Practical Recommendations

Michael BrenndoerferUpdated July 23, 20252 min read
Language AI HandbookAll chapters
Part 24 of 67BERT and Variants
Chapter 171BERT Fine-tuning › Practical Recommendations
Reading section 8 of 12 · Practical Recommendations · 2 minSection 8 of 12 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 171 of 513

Section 8 of 12 · 2 min read

BERT Fine-tuning › Practical RecommendationsOne section · Sections in this chapter8 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Successful fine-tuning requires balancing several factors. Here are concrete recommendations based on the original BERT paper and subsequent research.

Standard Recipe

For most classification and sequence labeling tasks:

Standard fine-tuning hyperparameters for BERT. These values work well across a wide range of tasks and dataset sizes.
ParameterRecommended Value
Learning rate2e-5 to 5e-5
Batch size16 or 32
Epochs2-4
Warmup ratio10% of total steps
Weight decay0.01
Max sequence length128-512 (task-dependent)
Dropout0.1

Dataset Size Guidelines

The amount of training data affects hyperparameter choices:

  • Small datasets (< 1K examples): Use lower learning rates (1e-5), more epochs (4-5), and consider layer freezing
  • Medium datasets (1K-10K examples): Standard settings work well
  • Large datasets (> 10K examples): Can use larger learning rates (5e-5), fewer epochs (2-3)

Task-Specific Considerations

Different tasks benefit from different approaches:

Classification: Standard recipe works well. Focus on class imbalance if present.

Sequence labeling (NER, POS): Handle subword alignment carefully. Consider CRF layer on top for structured prediction.

Question answering: Use longer max sequence lengths (384-512). Handle impossible questions by allowing null predictions.

Sentence pair tasks: Use segment embeddings. Consider whether task requires symmetric (similarity) or asymmetric (entailment) modeling.

When Fine-tuning Fails

If results are poor, try these debugging steps:

  1. Check data quality: Are labels correct? Is the task well-defined?
  2. Try different learning rates: Run a sweep from 1e-5 to 5e-5
  3. Increase training data: Can you augment or synthesize examples?
  4. Use a different model size: BERT-Large may help for complex tasks
  5. Consider domain mismatch: Pre-train further on domain-specific data first