Embedding Models: Architecture, Pooling & Selection › From Token Embeddings to Sentence Embeddings

As covered in previous chapters, encoder models like BERT produce contextual token embeddings: one vector per input token, where each vector is influenced by the surrounding context through self-attention. A sentence of 12 tokens passed through BERT yields 12 vectors of dimension 768. But for retrieval, we need one vector per document chunk. The bridge between token-level representations and a single sentence-level vector is what defines an embedding model, and this bridge is anything but trivial.

To appreciate why this distillation is non-trivial, consider the nature of the problem more carefully. A transformer encoder processes text through multiple layers of self-attention and feed-forward transformations, creating at each layer a set of contextualized representations, one per token. By the final layer, each token's hidden state is richly informed by all the other tokens in the input. The vector for "bank" in "the river bank was muddy" encodes the word "bank" and the fact that it is a physical location near water rather than a financial institution, because self-attention has allowed the model to blend contextual information from all surrounding tokens into each position's representation. These representations are extraordinarily rich, but they remain anchored to individual token positions. There is no built-in mechanism in a standard transformer encoder that automatically distills an entire sequence's meaning into a single, compact vector. That distillation is precisely what an embedding model must accomplish, and the method it uses to do so has a significant impact on retrieval quality.

The key insight is that collapsing 12 token vectors into 1 sentence vector is fundamentally a lossy compression problem. You cannot preserve every nuance of every token's contextual representation in a single fixed-length vector. The question is: which information should you keep? Different pooling strategies make different choices about this compression, and the right choice depends on both the model architecture and the training objective. A pooling strategy is a design choice that the model must be trained to exploit, rather than a mathematical operation applied only after training. The model's weights, shaped by the training loss, determine what information ends up in each position's hidden state. If the model was trained assuming mean pooling would be used, the hidden states will encode information in a form optimized for averaging. If you then apply CLS pooling, you're reading information out of a representation that was never designed to concentrate meaning at the CLS position, and quality degrades.

The naive approach of simply using a pre-trained BERT model to generate sentence embeddings performs surprisingly poorly. Reimers and Gurevych (2019) demonstrated that averaging BERT's token embeddings without further training produces representations that are often worse than GloVe averages for semantic similarity tasks. This finding was surprising at the time because BERT represented a massive leap forward in NLP performance on almost every other benchmark. The problem is that BERT's pre-training objectives (masked language modeling and next sentence prediction) optimize for token-level predictions, not for creating globally meaningful sentence vectors. BERT learns to predict missing words and to decide whether two sentences follow each other in the original document, but neither of these tasks requires the model to compress the overall meaning of a sentence into a single point in vector space. The token-level hidden states carry abundant contextual information, but that information is distributed across positions in a way that does not naturally aggregate into a coherent whole without explicit guidance.

This insight motivated the development of dedicated embedding models: transformer encoders that are fine-tuned with objectives that explicitly shape the geometry of the output space. The contrastive learning techniques from the previous chapter are precisely how this fine-tuning is done. By training the model to push embeddings of semantically similar texts closer together while pulling embeddings of dissimilar texts apart, the fine-tuning process turns a collection of token-level representations into a structured embedding space where distances encode semantic similarity. The pre-trained transformer provides a strong foundation of linguistic knowledge; the contrastive fine-tuning teaches the model how to organize that knowledge into a space that supports retrieval.