BERT Representations › Summary

BERT produces rich contextual representations, but extracting the right representation for your task requires careful choices. This chapter walked through every major dimension of that choice: which token to use, which layer to draw from, how to pool across the sequence, and how much of the model to update during training.

The [CLS] token provides a convenient sentence representation, learned through next sentence prediction during pre-training. It works well for classification tasks that are structurally similar to NSP, but performs poorly for semantic similarity because NSP didn't train the embedding space to be geometrically well-suited for cosine comparison. When similarity matters, use mean pooling or fine-tune with contrastive objectives.

Different layers encode different linguistic properties. Lower layers capture syntax and POS information, while upper layers capture semantics and task-relevant features. This hierarchy isn't a quirk of BERT's architecture; it reflects the general principle that deep networks build increasingly abstract representations. Layer combination strategies like concatenation, weighted sums, or scalar mixing let you capture information across the full hierarchy, and the ELMo-inspired ScalarMix module makes this combination learnable with only a handful of additional parameters.

Pooling strategies aggregate token representations into sentence vectors. Mean pooling often outperforms [CLS] for similarity tasks because it distributes the representational burden across all content tokens rather than relying on a single position. Max pooling captures the most salient signals per dimension, which benefits keyword-sensitive tasks. Attention-weighted pooling uses the model's own learned importance scores, though the practical gains over mean pooling are often modest.

The frozen versus fine-tuned decision depends on data size, compute budget, and domain match. With limited data, frozen features plus simple classifiers often work as well as fine-tuning, and avoid the risk of catastrophic forgetting. With abundant data, fine-tuning can improve performance but requires more compute and careful regularization. Partial fine-tuning, whether through layer freezing or gradual unfreezing, offers a practical middle ground for the gray zone between these extremes.

The anisotropy problem, context length limits, domain mismatch, and computational cost are the main limitations to plan around. Each has known mitigations: contrastive fine-tuning for anisotropy, hierarchical chunking for long documents, domain-adaptive pre-training for domain mismatch, and model distillation for latency.

Understanding these representation choices lets you get more out of BERT without blindly defaulting to fine-tuning. Sometimes the simplest approach, frozen features with mean pooling, is exactly what your task needs. The decision framework and linear probe technique give you a principled way to evaluate options quickly before committing to a full training pipeline.