BERT Representations › BERT Representations

You've trained or downloaded a BERT model. Now what? The raw transformer produces 12 or 24 layers of hidden states, each containing one 768-dimensional vector per token. That's a lot of numbers. How do you turn them into something useful for your downstream task?

Think of BERT as a very sophisticated translator, one that doesn't translate between human languages but between raw text and dense numerical meaning. Every word it reads gets encoded not in isolation but in the context of every surrounding word. After 12 layers of bidirectional attention and feed-forward transformation, BERT has embedded each token inside a 768-dimensional space where geometry encodes meaning. The sentence "The bank approved the loan" results in a vector for "bank" that points toward financial meaning, while the same word in "She sat by the river bank" points somewhere quite different. This contextual sensitivity is exactly what makes BERT representations so valuable.

The challenge is that BERT produces far too many vectors to use directly. A sentence of 20 tokens produces 20 vectors from each of 12 layers, giving you 240 vectors of dimension 768. Most downstream tasks need a single fixed-size representation. The extraction strategy you choose determines how much of BERT's knowledge flows through to your application.

The key insight is that different extraction choices answer different questions. Should you use the final layer, or would an intermediate layer serve better? Should you use the special [CLS] token that was designed to represent the whole sequence, or should you pool across all token positions? Should you freeze BERT's weights and treat it as a static feature extractor, or should you fine-tune the entire model on your labeled data? None of these questions has a universal answer. The best choice depends on your task, your data size, your computational budget, and whether your domain matches BERT's pre-training data.

In practice, these decisions matter enormously. Teams building semantic search systems have found that using the wrong pooling strategy can cut retrieval accuracy in half compared to the right one. Researchers doing syntactic analysis have discovered that intermediate layers outperform the final layer for certain probing tasks. Practitioners with tiny labeled datasets have seen frozen features outperform full fine-tuning. The representation extraction layer is where a lot of real-world NLP performance is won or lost.

This chapter tackles the representation extraction problem systematically. We'll explore the [CLS] token, why it works for some tasks and fails for others, and which layers contain the most useful information. We'll compare pooling strategies, examine when to freeze representations versus fine-tune, and build intuition for choosing the right approach. By the end, you'll know how to extract meaningful representations from BERT for classification and retrieval tasks, including similarity search.