BERT Representations › Frozen vs Fine-Tuned Representations
Language AI HandbookAll chapters BERT Representations › Frozen vs Fine-Tuned RepresentationsOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The choice between frozen features and fine-tuning depends on your data, compute budget, and task requirements. Let's examine the trade-offs.
The core tension is between generalization and specialization. BERT's pre-trained weights represent a highly general model of English language structure, trained on a massive and diverse corpus. Those weights encode broad linguistic knowledge that transfers well across tasks. Fine-tuning updates those weights to emphasize task-specific patterns, which can dramatically improve performance when you have sufficient labeled data. But with limited data, the updates can be noisy: BERT's general knowledge gets overwritten by noise from a small, potentially unrepresentative sample.
Think of it this way. Frozen BERT is like a well-traveled generalist who has read everything but knows nothing specific about your industry. They can still provide useful analysis of your documents because they encode broad linguistic patterns. Fine-tuned BERT is like sending that generalist through a specialized training program on your domain. With enough training data and good curriculum design, they become a domain expert. With too little data or poor-quality examples, they forget some of their general knowledge without learning enough about the domain.
In practice, the crossover point, where fine-tuning starts outperforming frozen features, varies by task and domain. For most standard NLP benchmarks with thousands of training examples, fine-tuning wins by a meaningful margin. For low-resource scenarios, niche domains with little annotated data, or rapid prototyping situations, frozen features are the pragmatic choice.
When to Use Frozen Representations
Frozen representations work well when:
- Data is limited: With fewer than 1000 examples, fine-tuning risks overfitting. The pre-trained features are already powerful
- Compute is constrained: No GPU access, or limited training time
- Exploration phase: You're testing many different approaches quickly
- Features are reused: You'll try many classifiers on the same embeddings
With very limited data (100-500 examples), frozen features often match or beat fine-tuning. The pre-trained representations often transfer well, while fine-tuning on tiny datasets can cause the model to overfit or forget useful pre-trained knowledge. Notice that the lines cross around 500-1,000 examples in the chart above. Below that threshold, frozen features benefit from BERT's massive pre-training without the risk of catastrophic forgetting on a small, noisy labeled set.
When to Fine-Tune
Fine-tuning becomes advantageous when:
- Data is plentiful: Thousands of labeled examples allow the model to adapt without overfitting
- Task differs from pre-training: BERT was trained on Wikipedia and books. Your domain (legal, medical, code) may require adaptation
- Maximum performance matters: Squeezing out every percentage point of accuracy justifies the compute cost
- Representations need task-specific adjustment: The optimal features for your task may differ from generic language understanding
The domain mismatch issue deserves special attention. BERT's Wikipedia and BookCorpus training data uses formal, edited prose. If your application processes Twitter posts, legal contracts, clinical notes, or source code, BERT's representations may be systematically poor for your vocabulary and syntax. In these situations, domain-adaptive pre-training, which involves continuing BERT's MLM pre-training on in-domain unlabeled text before task-specific fine-tuning, has been shown to improve performance substantially. SciBERT applies this approach to scientific literature, while BioBERT and LegalBERT do so for biomedical and legal text.
Partial Fine-Tuning Strategies
You don't have to choose between fully frozen and fully fine-tuned. Intermediate strategies often work well.
The motivation for partial fine-tuning comes from two observations. First, probing studies show that lower BERT layers encode fairly universal linguistic properties that transfer well across tasks. It seems wasteful to update weights that already capture core linguistic structure, from syntax and morphology to basic semantics. Second, fine-tuning all layers simultaneously on limited data creates a risk that the lower layers' stable, general representations get destabilized by gradient noise from a small task-specific dataset. Keeping early layers frozen preserves the foundation while allowing later layers to adapt.
Freeze early layers, fine-tune later layers:
def freeze_layers(model, num_layers_to_freeze):
"""
Freeze the embedding layer and first N transformer layers.
Later layers remain trainable.
"""
# Freeze embeddings
for param in model.embeddings.parameters():
param.requires_grad = False
# Freeze specified encoder layers
for i, layer in enumerate(model.encoder.layer):
if i < num_layers_to_freeze:
for param in layer.parameters():
param.requires_grad = False
# Example: freeze first 8 layers, fine-tune layers 9-12
model_partial = BertModel.from_pretrained("bert-base-uncased")
freeze_layers(model_partial, num_layers_to_freeze=8)Partial Fine-Tuning (Freeze First 8 Layers) ---------------------------------------- Total parameters: 109,482,240 Frozen parameters: 80,540,160 (73.6%) Trainable parameters: 28,942,080 (26.4%)
Freezing early layers preserves the general linguistic features BERT learned during pre-training. Fine-tuning later layers allows task-specific adaptation where it matters most. In practice, freezing the first 6 to 8 of BERT-base's 12 layers and fine-tuning the remaining layers is a common starting configuration. This approach also reduces memory requirements because frozen parameters don't need gradient storage, allowing you to fit larger batches on the same hardware.
Gradual unfreezing starts fully frozen and progressively unfreezes layers during training:
class GradualUnfreezer:
"""
Progressively unfreeze layers during training.
Start with only the classifier trainable, then unfreeze
BERT layers from top to bottom.
"""
def __init__(self, model, total_epochs, unfreeze_per_epoch=2):
self.model = model
self.total_epochs = total_epochs
self.unfreeze_per_epoch = unfreeze_per_epoch
self.num_layers = len(model.encoder.layer)
# Start fully frozen
for param in model.parameters():
param.requires_grad = False
def unfreeze_step(self, epoch):
"""Unfreeze layers based on current epoch."""
layers_to_unfreeze = min(
epoch * self.unfreeze_per_epoch, self.num_layers
)
# Unfreeze from the top (last layers first)
for i in range(
self.num_layers - 1, self.num_layers - 1 - layers_to_unfreeze, -1
):
if i >= 0:
for param in self.model.encoder.layer[i].parameters():
param.requires_grad = True
return layers_to_unfreeze
# Example usage
unfreezer = GradualUnfreezer(
model_partial, total_epochs=6, unfreeze_per_epoch=2
)Gradual Unfreezing Schedule ---------------------------------------- Epoch 0: 0 layers unfrozen, 0 trainable params Epoch 1: 2 layers unfrozen, 14,175,744 trainable params Epoch 2: 4 layers unfrozen, 28,351,488 trainable params Epoch 3: 6 layers unfrozen, 42,527,232 trainable params Epoch 4: 8 layers unfrozen, 56,702,976 trainable params Epoch 5: 10 layers unfrozen, 70,878,720 trainable params
Gradual unfreezing lets the classifier head stabilize before the BERT layers start changing. This can lead to more stable training, especially with limited data. The technique was popularized by the ULMFiT paper (2018), which showed that gradual unfreezing combined with discriminative learning rates, where lower layers use smaller learning rates than upper layers, could adapt a language model to new tasks very efficiently. The intuition is that lower layers should change slowly because they encode fundamental knowledge, while upper layers can change more aggressively to fit the new task distribution.
Representation Quality Metrics
How do you know if frozen representations are "good enough" for your task? Several metrics can help.
Linear probe accuracy measures how well a linear classifier can use the features:
def linear_probe_score(features, labels, cv=5):
"""
Evaluate representation quality via linear probe.
Higher accuracy means better features for the task.
"""
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
clf = LogisticRegression(max_iter=1000, random_state=42)
scores = cross_val_score(clf, features, labels, cv=min(cv, len(labels)))
return scores.mean(), scores.std()
# Compare different layer representations
layer_scores = []
for layer_idx in [1, 4, 8, 12]: # Sample of layers
# Use a separate extractor for each layer
ext = BertFeatureExtractor(layer=layer_idx, pooling="mean")
feats = ext.extract(train_texts)
mean_score, std_score = linear_probe_score(feats, train_labels)
layer_scores.append((layer_idx, mean_score, std_score))Linear Probe Scores by Layer ---------------------------------------- Layer 1: 0.633 (+/- 0.306) Layer 4: 0.633 (+/- 0.306) Layer 8: 0.633 (+/- 0.306) Layer 12: 0.700 (+/- 0.267)
If linear probe accuracy is high, the frozen features already separate classes well. If it's low, fine-tuning might be necessary to reshape the representation space. A score above 0.85 for binary classification on frozen representations suggests that fine-tuning may only offer marginal gains. A score below 0.65 suggests that the representation space doesn't naturally separate your categories and that task-specific adaptation is likely worth the investment. Run this quick probe before committing to an expensive fine-tuning run; it costs almost nothing and can save hours of compute.