BERT Fine-tuning › The Fine-tuning Paradigm
Language AI HandbookAll chapters BERT Fine-tuning › The Fine-tuning ParadigmOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Pre-training teaches BERT general language understanding. Fine-tuning specializes that understanding for downstream tasks. The key insight is that most of BERT's learned representations transfer well across tasks, so only minor adjustments are needed.
The process of taking a pre-trained model and continuing training on task-specific data with a task-specific objective. Fine-tuning updates all or most of the model's parameters, adapting general representations to the target task while preserving useful pre-trained knowledge.
The fine-tuning workflow follows a consistent pattern:
- Load a pre-trained BERT model
- Add a task-specific head (classifier, token labeler, or span predictor)
- Train on labeled task data with a much lower learning rate than pre-training
- The entire model updates: both the new head and BERT's existing weights
This differs from feature extraction, where BERT's weights are frozen and only the task head trains. Fine-tuning typically achieves better performance because BERT can adapt its internal representations to the task, not just learn to use fixed features.
