BERT Representations › Limitations and Practical Considerations

BERT representations, whether frozen or fine-tuned, have important limitations to keep in mind. Understanding these limitations matters because each one corresponds to a real failure mode that practitioners encounter in production systems.

The anisotropy problem means that BERT's representation space isn't uniformly distributed. Embeddings cluster in a narrow cone, making cosine similarity less discriminative than you might expect. You saw this concretely in the PCA visualization earlier, where sentences from completely different topics all landed in a tightly packed region of the 2D projection. This happens because BERT's pre-training with masked language modeling doesn't encourage representations to spread uniformly across the embedding space. The model learns to produce representations that work well for predicting masked tokens, not for spreading sentences across a geometrically useful space. Methods like whitening, centering, or training with contrastive objectives (as in Sentence-BERT) can mitigate this issue. The SimCSE paper (2021) showed that even a simple approach of treating dropout as a form of data augmentation and training with a contrastive objective dramatically improves the geometric quality of BERT representations.

Context length restrictions cap BERT at 512 tokens. Longer documents require chunking, which breaks cross-chunk attention. For long documents, consider hierarchical approaches: encode chunks separately, then aggregate the chunk representations. This chunking strategy has practical consequences: a document that starts with an introduction stating the main claim and ends with a conclusion restating it will have those semantically related passages in separate chunks with no direct attention path between them. Models like Longformer and BigBird addressed this limitation with sparse attention patterns that allow much longer contexts.

Domain mismatch occurs when BERT's Wikipedia/BookCorpus training differs from your target domain. Legal documents, scientific papers, or social media text may benefit from domain-specific pre-training (LegalBERT, SciBERT, BERTweet) rather than vanilla BERT. The gap can be surprisingly large. BioBERT, trained on PubMed abstracts, significantly outperforms vanilla BERT on biomedical NER tasks not because it's a fundamentally different model, but because it has seen the specialized vocabulary and syntactic patterns of scientific writing. If your application deals with domain-specific text and BERT's performance is disappointing, the first thing to try is a domain-adapted variant.

Computational cost scales with sequence length squared due to attention. For production systems processing many requests, consider distilled models (DistilBERT) or cached frozen representations rather than running inference for every query. DistilBERT achieves roughly 97% of BERT's performance on GLUE benchmarks while running 60% faster and using 40% less memory. For use cases where sub-second latency matters and the slight accuracy trade-off is acceptable, distillation is a well-established solution.

Tokenization artifacts can affect representation quality in subtle ways. BERT's WordPiece tokenizer breaks rare words into subword pieces, so "anticonstitutionally" might become multiple tokens. The representations of these subword pieces are averaged in mean pooling, which works reasonably well for content words. But for tasks that require precise morphological analysis, the subword granularity can introduce noise. Be aware of how your tokenizer handles domain-specific terminology, especially in technical or scientific domains where specialized compound terms are common.

Despite these limitations, BERT representations remain a strong foundation for most NLP tasks. The key is matching your extraction strategy to your specific constraints and requirements. Every limitation listed here has a known mitigation strategy, and the research community has spent years developing better alternatives. Understanding what BERT can't do well helps you know when to reach for those alternatives.

When working with BERT representations, several parameters significantly impact the quality and utility of extracted features:

• layer: Which transformer layer to extract representations from. Use -1 (last layer) for most tasks, but consider middle layers (5-8) for syntactic tasks like POS tagging or dependency parsing. The last layer contains the most task-adapted representations after fine-tuning
• pooling: Strategy for aggregating token representations into sentence vectors. Options include "cls" (use only the [CLS] token), "mean" (average all tokens), or "max" (element-wise maximum). Mean pooling typically performs better for similarity tasks
• output_hidden_states: Set to True when calling the model to access all layer representations. Required for layer selection or layer combination strategies
• output_attentions: Set to True to access attention weights for attention-weighted pooling. Adds computational overhead
• max_length: Maximum sequence length for tokenization (default 512 for BERT). Longer sequences are truncated, potentially losing important context
• num_layers_to_freeze: When partially fine-tuning, the number of early layers to keep frozen. Typically freeze 6-10 layers to preserve general linguistic knowledge while allowing task-specific adaptation in upper layers
• batch_size: Number of texts to process simultaneously during feature extraction. Larger batches are faster but require more memory. Adjust based on available GPU memory