BERT Contextual Embeddings: Extraction and Use

You've trained or downloaded a BERT model. Now what? The raw transformer produces 12 or 24 layers of hidden states, each containing one 768-dimensional vector per token. That's a lot of numbers. How do you turn them into something useful for your downstream task?

Think of BERT as a very sophisticated translator, one that doesn't translate between human languages but between raw text and dense numerical meaning. Every word it reads gets encoded not in isolation but in the context of every surrounding word. After 12 layers of bidirectional attention and feed-forward transformation, BERT has embedded each token inside a 768-dimensional space where geometry encodes meaning. The sentence "The bank approved the loan" results in a vector for "bank" that points toward financial meaning, while the same word in "She sat by the river bank" points somewhere quite different. This contextual sensitivity is exactly what makes BERT representations so valuable.

The challenge is that BERT produces far too many vectors to use directly. A sentence of 20 tokens produces 20 vectors from each of 12 layers, giving you 240 vectors of dimension 768. Most downstream tasks need a single fixed-size representation. The extraction strategy you choose determines how much of BERT's knowledge flows through to your application.

The key insight is that different extraction choices answer different questions. Should you use the final layer, or would an intermediate layer serve better? Should you use the special [CLS] token that was designed to represent the whole sequence, or should you pool across all token positions? Should you freeze BERT's weights and treat it as a static feature extractor, or should you fine-tune the entire model on your labeled data? None of these questions has a universal answer. The best choice depends on your task, your data size, your computational budget, and whether your domain matches BERT's pre-training data.

In practice, these decisions matter enormously. Teams building semantic search systems have found that using the wrong pooling strategy can cut retrieval accuracy in half compared to the right one. Researchers doing syntactic analysis have discovered that intermediate layers outperform the final layer for certain probing tasks. Practitioners with tiny labeled datasets have seen frozen features outperform full fine-tuning. The representation extraction layer is where a lot of real-world NLP performance is won or lost.

This chapter tackles the representation extraction problem systematically. We'll explore the [CLS] token, why it works for some tasks and fails for others, and which layers contain the most useful information. We'll compare pooling strategies, examine when to freeze representations versus fine-tune, and build intuition for choosing the right approach. By the end, you'll know how to extract meaningful representations from BERT for classification and retrieval tasks, including similarity search.

BERT prepends a special [CLS] token to every input sequence. After passing through all transformer layers, this token's final hidden state is intended to represent the entire sequence. But why does shoving extra information into position zero work at all?

The key is self-attention. Every layer allows [CLS] to attend to all other tokens. Over 12 layers, information from the entire sequence flows into this position. The model learns during pre-training, through the Next Sentence Prediction task, to aggregate sentence-level meaning at [CLS].

Think of [CLS] as a reporter who attends every conversation. The reporter can listen to all participants simultaneously, ask follow-up questions across the whole room, and gradually builds a mental summary that captures the overall theme of the discussion. By the time the conversation ends, the reporter's notes reflect the gist of everything said. That summary is the [CLS] vector. Of course, a reporter's summary emphasizes some things and ignores others, and a summary is always a lossy compression of the original. The same is true for [CLS].

Notice that the [CLS] token starts with no information about the sentence. At the embedding layer, it holds only a learned position-zero embedding that BERT associated with sentence classification during pre-training. Its final representation comes entirely from what it learned to absorb through attention over all 12 layers. This is a learned compression process: BERT's pre-training rewarded the [CLS] vector whenever it could correctly predict whether two sentences were consecutive. That training signal shaped the [CLS] representation to encode sentence-level coherence and topic.

Let's extract the [CLS] representation from a pre-trained BERT model.

The histogram reveals that most embedding dimensions have values between -0.5 and 0.5, with a roughly symmetric distribution. This structure emerges from BERT's layer normalization, which ensures stable training by keeping activations in a controlled range.

The [CLS] representation works well for tasks that mirror BERT's pre-training objectives. Classification tasks, especially binary ones, align naturally with how [CLS] was trained through NSP.

In practice, the [CLS] token is the default choice for text classification pipelines built on BERT. When you fine-tune BERT for sentiment analysis, topic classification, or natural language inference, you typically attach a linear layer to the [CLS] output and train the whole system end-to-end. The combination works because gradient descent can reshape the [CLS] representation to separate classes, building on the general aggregation capacity the model developed during pre-training. The pre-training doesn't hand you a perfect sentence embedding; it gives you a vector that is already structured to receive a classification signal, which makes fine-tuning efficient.

Even without fine-tuning, the [CLS] representations show some clustering by sentiment. Positive texts are more similar to each other than to negative texts. This separation, while modest, demonstrates that [CLS] captures semantic information relevant to classification.

The [CLS] token has a critical limitation: it was trained for NSP, not for general semantic similarity. For tasks like semantic search or sentence similarity, raw [CLS] embeddings perform surprisingly poorly.

The NSP objective is a weak training signal. It asks whether sentence B could plausibly follow sentence A in a real document. This is a binary prediction task requiring only that the model notice topical coherence, not that it represents the full meaning of each sentence in a geometrically useful way. A [CLS] vector trained on NSP learns to encode "this pair is coherent" or "this pair is random," but it doesn't learn to place semantically similar sentences near each other in vector space. When you compute cosine similarity between two [CLS] vectors, you're measuring something that correlates weakly at best with human judgments of sentence similarity.

Notice that this failure isn't a bug in BERT; it's a consequence of using a representation for a purpose it wasn't trained for. The model did exactly what it was asked to do during pre-training. The problem is that NSP's training signal didn't require the embedding space to have the geometric properties that similarity tasks need. This is why evaluating your representation strategy on a small held-out set before committing to a production pipeline is so important.

The PCA projection makes the practical limitation visible: topic labels intermix instead of forming compact, separated clusters. The first two principal components summarize only part of the 768-dimensional variation, so this view is not a complete map of the embedding space. What it does show is that these raw [CLS] vectors do not separate the example topics cleanly.

This is why models like Sentence-BERT (SBERT) exist. They fine-tune BERT specifically for sentence similarity using contrastive or siamese objectives, producing representations where cosine similarity correlates with semantic relatedness. We'll examine SBERT in more depth in the chapter on dense retrieval and semantic search. For now, the takeaway is clear: if your task involves comparing sentences by meaning rather than classifying them, don't reach for raw [CLS] embeddings.

BERT doesn't produce just one representation. BERT-base has 12 transformer layers, each outputting a full sequence of hidden states. Which layer should you use?

The answer depends on your task. Different layers capture different types of linguistic information. A rich body of research on "probing classifiers" has shed light on what each layer knows. The idea is simple: train a small linear classifier to predict a linguistic property, such as part-of-speech tags or syntactic dependencies, using only the representations from a single BERT layer. The layer where the linear classifier performs best is the layer that encodes that property most cleanly. Across many such probing studies, a consistent pattern has emerged:

• Lower layers (1-4) capture surface-level features like part-of-speech and simple syntax
• Middle layers (5-8) capture syntactic structures and dependencies
• Upper layers (9-12) capture task-specific and semantic information

Think of the layers as a hierarchy of abstraction, similar to what happens in convolutional neural networks applied to images. Early CNN layers detect edges and textures; later layers detect faces and objects. In BERT, early layers learn that "running" is a verb and "the" is a determiner, while later layers learn that "the bank" in a financial context means something fundamentally different from "the bank" in a geological context.

The key insight is that BERT's layers are not interchangeable. Using the last layer by default is a reasonable starting point, but it is not always the best choice. For tasks that depend on surface form or morphology, such as named entity recognition in a highly inflected language, representations from layers 3 to 5 may carry more signal than the final layer. For tasks that require deep world knowledge and semantic reasoning, layers 10 to 12 tend to perform better. The right layer is a hyperparameter worth tuning.

Let's visualize how representations evolve across layers.

The heatmap reveals a clear pattern. Adjacent layers are highly similar, shown by the bright diagonal band. But layer 1 and layer 12 are quite different, with similarity around 0.5. The information encoded at different depths varies substantially.

Different NLP tasks benefit from different layers. Probing experiments have systematically tested which layers encode which linguistic properties.

POS tagging peaks at layers 3-4, while coreference resolution benefits from the deepest layers. This has practical implications. If you're building a part-of-speech tagger, using the last layer might hurt performance. Using layer 3 or 4 could give you a better starting point.

Why limit yourself to one layer? Several strategies combine information across layers.

The motivation for combining layers comes directly from the probing results. If syntactic information peaks at layer 6 and semantic information peaks at layer 11, a task that requires both, such as semantic role labeling, which asks which arguments fill which roles in a sentence, might benefit from using both. Rather than picking one layer and discarding the others, you can fuse information across the stack. This idea predates BERT: ELMo (2018) combined representations from two LSTM layers using learned weights, achieving state-of-the-art results on a wide range of tasks by letting downstream models decide how much to weight each level of abstraction.

In practice, the last four layers are the most commonly combined because they carry the richest semantic signal and are most likely to help with downstream tasks. But for structured prediction tasks like dependency parsing, combining a broader range of layers can capture both low-level morphological features and high-level syntactic patterns in a single representation.

Concatenation stacks representations from multiple layers:

ScalarMix adds only 14 parameters (13 mixing weights plus gamma), making it efficient to learn the optimal layer combination during fine-tuning.

The [CLS] token is one way to get a sentence representation, but it's not the only way. Pooling strategies aggregate information across all token positions.

The underlying question pooling answers is: given that every token in the sequence has a rich contextual representation, how should we combine them into a single fixed-size vector? Each token's representation already encodes context from the entire sequence through self-attention, so the representations aren't independent. A pooled representation tries to summarize this distributed information into something compact and task-relevant.

Notice that the [CLS] approach is itself a pooling strategy of sorts, one where all the aggregation work happens inside the transformer through attention, and only the special token's output is used. The alternative pooling methods described here perform the aggregation after the transformer, using simple arithmetic over the output token representations. Both approaches have their place, and the best choice depends on what you want the representation to capture.

In practice, when building systems that need to compare sentence meanings, such as question-answering retrievers or document similarity engines, mean pooling over the final layer often outperforms [CLS] without any additional fine-tuning. This is a quick win: simply change how you read the output tensor, and your similarity quality improves. The reason is that mean pooling spreads the representation load across all content tokens rather than concentrating it at a single position that was not specifically trained for generic similarity.

Mean pooling averages the representations of all tokens (typically excluding special tokens).

Mean pooling gives equal weight to every token. This works well when all parts of the sentence contribute equally to the overall meaning, which is often the case for similarity tasks. The attention mask is essential here: without it, padding tokens would dilute the representation by contributing zero-content vectors. Always pass the mask when pooling to ensure that only real tokens contribute to the average.

Max pooling takes the maximum value across the sequence for each dimension.

Max pooling captures the strongest signals in each dimension. It can be useful when specific keywords or phrases are most important, though it tends to be more sensitive to outliers than mean pooling. The intuition is that for some tasks, presence matters more than prevalence. If a document mentions "cancer" once in 500 tokens, mean pooling dilutes that signal, while max pooling preserves the highest activation in the "cancer-related" dimensions regardless of how rarely the topic appeared. For topic labeling and keyword-sensitive retrieval, this property can be an advantage.

Let's compare how different pooling strategies handle the same sentences.

A more sophisticated approach uses the model's own attention weights to determine token importance.

Attention-weighted pooling lets the model decide which tokens matter most. Tokens that receive more attention across the sequence get higher weight in the final representation. In this example, content-bearing words like "bank", "approved", and "loan" naturally receive higher importance scores than function words like "the".

The appeal of attention-weighted pooling is that it's data-driven rather than heuristic. Rather than uniformly averaging or blindly taking the maximum, the weights come from the model's own learned attention patterns. Notice, however, that this approach has a subtle limitation: attention weights reflect what the model found important for its pre-training objective, which may not align perfectly with what's important for your task. A model trained on next-sentence prediction might attend heavily to discourse markers and topic words, but for a technical document retrieval task, precise terminology might matter more. In practice, attention-weighted pooling adds engineering complexity without reliably outperforming mean pooling on most benchmarks, so treat it as a specialized tool rather than a default.

Using BERT as a feature extractor means taking its representations as fixed inputs to a downstream model. This contrasts with fine-tuning, where BERT's weights are updated during training.

Think of the feature extraction approach as hiring a skilled translator who has spent years studying a language. Instead of teaching them about your specific business domain, you ask them to convert every document into a standardized format, and then you build your analysis on top of their translations. The translator's skills are fixed; what you're customizing is the analysis layer, not the translation.

The historical motivation for feature extraction comes from the era before BERT, when contextual embeddings like ELMo were computationally expensive to run on large datasets. ELMo's original paper described using it primarily as a feature extractor: compute representations once, cache them, and feed them to task-specific models. BERT changed the economics of fine-tuning by making the full model accessible to practitioners with moderate GPU resources, but feature extraction remains valuable when inference latency and compute budget are constraints.

In practice, feature extraction is the right default when you're exploring a new task quickly. You can compare logistic regression, SVM, gradient boosting, and neural classifiers on the same BERT features in minutes, cycling through hypotheses without waiting for full model fine-tuning runs. Once you've identified a promising direction, you can decide whether fine-tuning on top of that approach is worth the additional investment.

Feature extraction treats BERT as a frozen encoder. You pass text through BERT once, save the embeddings, and train a separate classifier on those embeddings.

These 768-dimensional features can now feed into any classifier: logistic regression, SVM, random forest, or a simple neural network. The most important property to note is that the features are precomputed. Once you've run BERT over your corpus and saved the output embeddings to disk, subsequent training runs don't require a GPU at all. You're just fitting a classifier to a matrix of numbers, which even a laptop CPU handles easily. This separation of the expensive inference step from the cheap classification step is the core practical advantage of the feature extraction pipeline.

Let's train a simple classifier using extracted BERT features.

Even with a tiny dataset, the classifier achieves reasonable accuracy because BERT's pre-trained features already encode rich semantic information. The BERT encoder has seen hundreds of millions of sentences and learned that "I loved this" and "Absolutely wonderful" point toward similar regions of embedding space. The linear classifier's job is much easier than learning from scratch: it just needs to find the hyperplane that separates these pre-organized clusters.

Using BERT as a frozen feature extractor offers several practical benefits:

• Speed: Extract features once, then train classifiers instantly. You don't need GPU access for every experiment
• Simplicity: Standard ML pipelines work directly. No need for gradient-based optimization of large models
• Low resource: Train on CPU with minimal memory. Fine-tuning BERT requires significant GPU memory
• Interpretability: Downstream models can be simpler and more interpretable (e.g., logistic regression with feature importance)

The choice between frozen features and fine-tuning depends on your data, compute budget, and task requirements. Let's examine the trade-offs.

The core tension is between generalization and specialization. BERT's pre-trained weights represent a highly general model of English language structure, trained on a massive and diverse corpus. Those weights encode broad linguistic knowledge that transfers well across tasks. Fine-tuning updates those weights to emphasize task-specific patterns, which can dramatically improve performance when you have sufficient labeled data. But with limited data, the updates can be noisy: BERT's general knowledge gets overwritten by noise from a small, potentially unrepresentative sample.

Think of it this way. Frozen BERT is like a well-traveled generalist who has read everything but knows nothing specific about your industry. They can still provide useful analysis of your documents because they encode broad linguistic patterns. Fine-tuned BERT is like sending that generalist through a specialized training program on your domain. With enough training data and good curriculum design, they become a domain expert. With too little data or poor-quality examples, they forget some of their general knowledge without learning enough about the domain.

In practice, the crossover point, where fine-tuning starts outperforming frozen features, varies by task and domain. For most standard NLP benchmarks with thousands of training examples, fine-tuning wins by a meaningful margin. For low-resource scenarios, niche domains with little annotated data, or rapid prototyping situations, frozen features are the pragmatic choice.

Frozen representations work well when:

• Data is limited: With fewer than 1000 examples, fine-tuning risks overfitting. The pre-trained features are already powerful
• Compute is constrained: No GPU access, or limited training time
• Exploration phase: You're testing many different approaches quickly
• Features are reused: You'll try many classifiers on the same embeddings

With very limited data (100-500 examples), frozen features often match or beat fine-tuning. The pre-trained representations often transfer well, while fine-tuning on tiny datasets can cause the model to overfit or forget useful pre-trained knowledge. Notice that the lines cross around 500-1,000 examples in the chart above. Below that threshold, frozen features benefit from BERT's massive pre-training without the risk of catastrophic forgetting on a small, noisy labeled set.

Fine-tuning becomes advantageous when:

• Data is plentiful: Thousands of labeled examples allow the model to adapt without overfitting
• Task differs from pre-training: BERT was trained on Wikipedia and books. Your domain (legal, medical, code) may require adaptation
• Maximum performance matters: Squeezing out every percentage point of accuracy justifies the compute cost
• Representations need task-specific adjustment: The optimal features for your task may differ from generic language understanding

The domain mismatch issue deserves special attention. BERT's Wikipedia and BookCorpus training data uses formal, edited prose. If your application processes Twitter posts, legal contracts, clinical notes, or source code, BERT's representations may be systematically poor for your vocabulary and syntax. In these situations, domain-adaptive pre-training, which involves continuing BERT's MLM pre-training on in-domain unlabeled text before task-specific fine-tuning, has been shown to improve performance substantially. SciBERT applies this approach to scientific literature, while BioBERT and LegalBERT do so for biomedical and legal text.

You don't have to choose between fully frozen and fully fine-tuned. Intermediate strategies often work well.

The motivation for partial fine-tuning comes from two observations. First, probing studies show that lower BERT layers encode fairly universal linguistic properties that transfer well across tasks. It seems wasteful to update weights that already capture core linguistic structure, from syntax and morphology to basic semantics. Second, fine-tuning all layers simultaneously on limited data creates a risk that the lower layers' stable, general representations get destabilized by gradient noise from a small task-specific dataset. Keeping early layers frozen preserves the foundation while allowing later layers to adapt.

Freeze early layers, fine-tune later layers:

Gradual unfreezing lets the classifier head stabilize before the BERT layers start changing. This can lead to more stable training, especially with limited data. The technique was popularized by the ULMFiT paper (2018), which showed that gradual unfreezing combined with discriminative learning rates, where lower layers use smaller learning rates than upper layers, could adapt a language model to new tasks very efficiently. The intuition is that lower layers should change slowly because they encode fundamental knowledge, while upper layers can change more aggressively to fit the new task distribution.

How do you know if frozen representations are "good enough" for your task? Several metrics can help.

Linear probe accuracy measures how well a linear classifier can use the features:

If linear probe accuracy is high, the frozen features already separate classes well. If it's low, fine-tuning might be necessary to reshape the representation space. A score above 0.85 for binary classification on frozen representations suggests that fine-tuning may only offer marginal gains. A score below 0.65 suggests that the representation space doesn't naturally separate your categories and that task-specific adaptation is likely worth the investment. Run this quick probe before committing to an expensive fine-tuning run; it costs almost nothing and can save hours of compute.

Let's synthesize everything into actionable guidelines.

Across the topics covered in this chapter, a few themes recur. First, representations extracted for one purpose often fail for a different purpose, even when both seem like "NLP tasks." The [CLS] token is great for classification but terrible for similarity. Layer 4 representations work better for syntax, while layer 11 works better for semantics. Mean pooling beats [CLS] for retrieval without any additional training. These aren't arbitrary preferences: each choice reflects a specific alignment, or misalignment, between how the representation was trained and what the downstream task needs.

Second, the amount of labeled data you have should drive the frozen-versus-fine-tuning decision more than any other factor. A practitioner with 200 annotated examples and a tight deadline should almost always start with frozen features. A practitioner with 50,000 annotated examples and access to a decent GPU should almost always fine-tune. The gray zone between those extremes, roughly 1,000 to 10,000 examples, is where partial fine-tuning strategies pay off most reliably.

Third, the representation extraction layer is a legitimate hyperparameter that deserves experimentation. Many practitioners skip straight to the last layer by default and never revisit the choice. Spending an hour running linear probes across a few representative layers can reveal whether you're leaving performance on the table.

Use this decision framework:

Quick reference table:

Avoid these mistakes when working with BERT representations:

• Using raw CLS for similarity: Vanilla BERT's [CLS] token wasn't trained for similarity. Use mean pooling or fine-tune with contrastive objectives
• Always using the last layer: For some tasks, middle layers work better. Experiment with layer selection
• Ignoring the attention mask: When pooling, always mask out padding tokens to avoid noise in your representations
• Fine-tuning with tiny data: With fewer than a few hundred examples, frozen features often outperform fine-tuning
• One-size-fits-all pooling: Match pooling to your task. CLS for classification, mean for similarity, token-level for NER

BERT representations, whether frozen or fine-tuned, have important limitations to keep in mind. Understanding these limitations matters because each one corresponds to a real failure mode that practitioners encounter in production systems.

The anisotropy problem means that BERT's representation space isn't uniformly distributed. Embeddings cluster in a narrow cone, making cosine similarity less discriminative than you might expect. You saw this concretely in the PCA visualization earlier, where sentences from completely different topics all landed in a tightly packed region of the 2D projection. This happens because BERT's pre-training with masked language modeling doesn't encourage representations to spread uniformly across the embedding space. The model learns to produce representations that work well for predicting masked tokens, not for spreading sentences across a geometrically useful space. Methods like whitening, centering, or training with contrastive objectives (as in Sentence-BERT) can mitigate this issue. The SimCSE paper (2021) showed that even a simple approach of treating dropout as a form of data augmentation and training with a contrastive objective dramatically improves the geometric quality of BERT representations.

Context length restrictions cap BERT at 512 tokens. Longer documents require chunking, which breaks cross-chunk attention. For long documents, consider hierarchical approaches: encode chunks separately, then aggregate the chunk representations. This chunking strategy has practical consequences: a document that starts with an introduction stating the main claim and ends with a conclusion restating it will have those semantically related passages in separate chunks with no direct attention path between them. Models like Longformer and BigBird addressed this limitation with sparse attention patterns that allow much longer contexts.

Domain mismatch occurs when BERT's Wikipedia/BookCorpus training differs from your target domain. Legal documents, scientific papers, or social media text may benefit from domain-specific pre-training (LegalBERT, SciBERT, BERTweet) rather than vanilla BERT. The gap can be surprisingly large. BioBERT, trained on PubMed abstracts, significantly outperforms vanilla BERT on biomedical NER tasks not because it's a fundamentally different model, but because it has seen the specialized vocabulary and syntactic patterns of scientific writing. If your application deals with domain-specific text and BERT's performance is disappointing, the first thing to try is a domain-adapted variant.

Computational cost scales with sequence length squared due to attention. For production systems processing many requests, consider distilled models (DistilBERT) or cached frozen representations rather than running inference for every query. DistilBERT achieves roughly 97% of BERT's performance on GLUE benchmarks while running 60% faster and using 40% less memory. For use cases where sub-second latency matters and the slight accuracy trade-off is acceptable, distillation is a well-established solution.

Tokenization artifacts can affect representation quality in subtle ways. BERT's WordPiece tokenizer breaks rare words into subword pieces, so "anticonstitutionally" might become multiple tokens. The representations of these subword pieces are averaged in mean pooling, which works reasonably well for content words. But for tasks that require precise morphological analysis, the subword granularity can introduce noise. Be aware of how your tokenizer handles domain-specific terminology, especially in technical or scientific domains where specialized compound terms are common.

Despite these limitations, BERT representations remain a strong foundation for most NLP tasks. The key is matching your extraction strategy to your specific constraints and requirements. Every limitation listed here has a known mitigation strategy, and the research community has spent years developing better alternatives. Understanding what BERT can't do well helps you know when to reach for those alternatives.

When working with BERT representations, several parameters significantly impact the quality and utility of extracted features:

• layer: Which transformer layer to extract representations from. Use -1 (last layer) for most tasks, but consider middle layers (5-8) for syntactic tasks like POS tagging or dependency parsing. The last layer contains the most task-adapted representations after fine-tuning
• pooling: Strategy for aggregating token representations into sentence vectors. Options include "cls" (use only the [CLS] token), "mean" (average all tokens), or "max" (element-wise maximum). Mean pooling typically performs better for similarity tasks
• output_hidden_states: Set to True when calling the model to access all layer representations. Required for layer selection or layer combination strategies
• output_attentions: Set to True to access attention weights for attention-weighted pooling. Adds computational overhead
• max_length: Maximum sequence length for tokenization (default 512 for BERT). Longer sequences are truncated, potentially losing important context
• num_layers_to_freeze: When partially fine-tuning, the number of early layers to keep frozen. Typically freeze 6-10 layers to preserve general linguistic knowledge while allowing task-specific adaptation in upper layers
• batch_size: Number of texts to process simultaneously during feature extraction. Larger batches are faster but require more memory. Adjust based on available GPU memory

BERT produces rich contextual representations, but extracting the right representation for your task requires careful choices. This chapter walked through every major dimension of that choice: which token to use, which layer to draw from, how to pool across the sequence, and how much of the model to update during training.

The [CLS] token provides a convenient sentence representation, learned through next sentence prediction during pre-training. It works well for classification tasks that are structurally similar to NSP, but performs poorly for semantic similarity because NSP didn't train the embedding space to be geometrically well-suited for cosine comparison. When similarity matters, use mean pooling or fine-tune with contrastive objectives.

Different layers encode different linguistic properties. Lower layers capture syntax and POS information, while upper layers capture semantics and task-relevant features. This hierarchy isn't a quirk of BERT's architecture; it reflects the general principle that deep networks build increasingly abstract representations. Layer combination strategies like concatenation, weighted sums, or scalar mixing let you capture information across the full hierarchy, and the ELMo-inspired ScalarMix module makes this combination learnable with only a handful of additional parameters.

Pooling strategies aggregate token representations into sentence vectors. Mean pooling often outperforms [CLS] for similarity tasks because it distributes the representational burden across all content tokens rather than relying on a single position. Max pooling captures the most salient signals per dimension, which benefits keyword-sensitive tasks. Attention-weighted pooling uses the model's own learned importance scores, though the practical gains over mean pooling are often modest.

The frozen versus fine-tuned decision depends on data size, compute budget, and domain match. With limited data, frozen features plus simple classifiers often work as well as fine-tuning, and avoid the risk of catastrophic forgetting. With abundant data, fine-tuning can improve performance but requires more compute and careful regularization. Partial fine-tuning, whether through layer freezing or gradual unfreezing, offers a practical middle ground for the gray zone between these extremes.

The anisotropy problem, context length limits, domain mismatch, and computational cost are the main limitations to plan around. Each has known mitigations: contrastive fine-tuning for anisotropy, hierarchical chunking for long documents, domain-adaptive pre-training for domain mismatch, and model distillation for latency.

Understanding these representation choices lets you get more out of BERT without blindly defaulting to fine-tuning. Sometimes the simplest approach, frozen features with mean pooling, is exactly what your task needs. The decision framework and linear probe technique give you a principled way to evaluate options quickly before committing to a full training pipeline.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about BERT representations and how to extract them effectively.