Embedding Models: Architecture, Pooling & Selection › Limitations and Practical Considerations

Embedding models, despite their impressive capabilities, carry important limitations that every practitioner building RAG systems must understand. Recognizing these limitations allows you to design systems that compensate for them, to interpret retrieval failures correctly, and to set realistic expectations for system performance.

The most basic limitation is the information bottleneck. Compressing a passage of potentially hundreds of tokens into a single fixed-length vector inevitably loses information. A 768-dimensional vector, rich as it is, cannot encode every nuance of a 500-token passage. Important details that are not prominent in the aggregate representation, such as a specific date mentioned once in the middle of a paragraph, may not survive the compression process strongly enough to drive retrieval when a query asks specifically for that information. This is why cross-encoder rerankers, which perform full token-level cross-attention between the query and each candidate document, can substantially improve accuracy after first-stage embedding retrieval: they can recover fine-grained signals that were compressed away by the pooling operation. The two-stage architecture of coarse embedding retrieval followed by cross-encoder reranking is specifically designed to work around this bottleneck, and we will examine it in detail when we cover reranking later in this part.

Another significant limitation is domain mismatch. General-purpose embedding models are trained primarily on web text, Wikipedia, news articles, and common NLP benchmarks. When deployed on specialized domains such as medical literature, legal contracts, financial reports, or software codebases, their performance can drop substantially. The embedding space may not have learned the fine-grained distinctions that matter in your domain. A general model might place "myocardial infarction" and "cardiac arrest" close together in the embedding space because they are both heart-related medical conditions with overlapping vocabulary, even though they are clinically distinct conditions requiring different treatments and showing different pathologies. A medical embedding model trained on clinical text would learn to distinguish these concepts because the training data contains many examples where their differences matter. Domain-specific fine-tuning with contrastive learning on in-domain pairs can address this limitation, but it requires curated labeled data and training infrastructure.

There is also the issue of embedding drift over time. As better models become available or as your domain evolves, you may want to upgrade your embedding model. But all embeddings in your vector index must come from the same model: you cannot mix embeddings from different models because they lie in fundamentally different vector spaces where proximity has no consistent meaning across models. Upgrading your embedding model requires re-embedding the entire corpus and rebuilding the vector index from scratch. For large corpora, this can be a multi-day operation that consumes significant compute and storage and must be coordinated carefully to avoid downtime. This operational cost creates a form of technical debt: once you build a vector index with a particular model, switching models is expensive, which can deter you from adopting newer, better models.

Finally, embedding models inherit the biases present in their training data. If certain topics, perspectives, writing styles, or demographic groups are underrepresented in training, the embedding space will be poorly calibrated for inputs from those underrepresented categories. Queries from users asking about underrepresented topics, or queries phrased in language styles not well-represented in training data, may receive systematically worse retrieval results. This creates both fairness and reliability problems for applications serving diverse users or topics. Evaluating your embedding model across diverse query types and user populations, not just on the most common or representative cases, is important for understanding where these blind spots lie.