BERT Representations › Practical Recommendations
Language AI HandbookAll chapters BERT Representations › Practical RecommendationsOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's synthesize everything into actionable guidelines.
Across the topics covered in this chapter, a few themes recur. First, representations extracted for one purpose often fail for a different purpose, even when both seem like "NLP tasks." The [CLS] token is great for classification but terrible for similarity. Layer 4 representations work better for syntax, while layer 11 works better for semantics. Mean pooling beats [CLS] for retrieval without any additional training. These aren't arbitrary preferences: each choice reflects a specific alignment, or misalignment, between how the representation was trained and what the downstream task needs.
Second, the amount of labeled data you have should drive the frozen-versus-fine-tuning decision more than any other factor. A practitioner with 200 annotated examples and a tight deadline should almost always start with frozen features. A practitioner with 50,000 annotated examples and access to a decent GPU should almost always fine-tune. The gray zone between those extremes, roughly 1,000 to 10,000 examples, is where partial fine-tuning strategies pay off most reliably.
Third, the representation extraction layer is a legitimate hyperparameter that deserves experimentation. Many practitioners skip straight to the last layer by default and never revisit the choice. Spending an hour running linear probes across a few representative layers can reveal whether you're leaving performance on the table.
Choosing Your Approach
Use this decision framework:
Quick reference table:
| Scenario | Recommended Approach | Pooling | Layer |
|---|---|---|---|
| Text classification, limited data | Frozen + LogReg | Mean or CLS | Last |
| Semantic similarity | Frozen or fine-tuned | Mean | Last |
| Token-level tasks (NER) | Fine-tuned | None (use all tokens) | Last |
| Syntactic probing | Frozen | Token-level | Middle (5-8) |
| Production with latency constraints | Frozen + cache | Mean | Last |
Common Pitfalls
Avoid these mistakes when working with BERT representations:
- Using raw CLS for similarity: Vanilla BERT's
[CLS]token wasn't trained for similarity. Use mean pooling or fine-tune with contrastive objectives - Always using the last layer: For some tasks, middle layers work better. Experiment with layer selection
- Ignoring the attention mask: When pooling, always mask out padding tokens to avoid noise in your representations
- Fine-tuning with tiny data: With fewer than a few hundred examples, frozen features often outperform fine-tuning
- One-size-fits-all pooling: Match pooling to your task. CLS for classification, mean for similarity, token-level for NER