Vector Similarity Search: Metrics & Approximate Methods › Limitations and Practical Considerations

Vector similarity search is powerful but comes with important caveats that affect real-world RAG systems.

A key limitation is that vector similarity is not semantic equivalence. Two documents can have high cosine similarity without being relevant to the query, and relevant documents can sometimes have lower similarity scores than irrelevant ones. This happens because embedding models compress complex, fine-grained meaning into fixed-size vectors, and information is inevitably lost in this compression.

A query about "python programming" might retrieve documents about actual pythons if the embedding model does not sufficiently distinguish context. This is one reason why RAG pipelines typically include a reranking stage (covered later in this part), which uses a more expensive cross-encoder model to rescore the top candidates. The reranker examines the full query-document pair jointly, capturing cross-document context that a bi-encoder embedding model cannot represent. This two-stage pipeline, retrieve then rerank, is the standard architecture for production RAG.

The severity of the semantic gap depends on the quality and training data of the embedding model. A general-purpose embedding model trained on diverse web text will struggle with specialized technical domains where terminology overlaps with everyday language. Domain-specific fine-tuning of embedding models is one of the most effective ways to close this gap, and this is covered in the chapter on fine-tuning for retrieval.

The curse of dimensionality affects ANN algorithms in subtle but important ways. In very high-dimensional spaces, the ratio between the nearest and farthest neighbor distances shrinks. As $d$ grows, all pairwise distances converge toward the same value, which makes it increasingly difficult to distinguish between near and far neighbors.

This effect is most pronounced in uniformly distributed random data. Typical embedding dimensions (384-1536) are high enough that this effect is noticeable but not catastrophic. However, it means that ANN algorithms cannot offer the same theoretical guarantees in high dimensions that they can in low dimensions. In practice, the learned structure of real embeddings, which cluster by topic and style rather than being uniformly distributed, counteracts this effect, and ANN methods work much better on real data than worst-case theory would suggest.

The anisotropy of text embeddings, where vectors cluster in a narrow cone of the high-dimensional space rather than filling it uniformly, is both a curse and a blessing for search. It means that random pairs tend to have non-trivially positive cosine similarity (the curse), which makes it harder to distinguish relevant from irrelevant documents. But it also means that the clustering structure that ANN algorithms exploit is more pronounced, so cluster-based methods like IVF work particularly well.

Memory remains a persistent practical concern. Storing 100 million 768-dimensional vectors in float32 requires about 288 GB of RAM, before accounting for the additional memory needed by the index structure itself (graph edges for HNSW, centroid tables for IVF, etc.). Few single machines can accommodate this.

Product Quantization and other compression techniques, which we cover in a dedicated chapter, can reduce this by 10-30x, but at the cost of reduced recall. A production RAG system must meet its accuracy target within a fixed memory and latency budget. A common production architecture uses PQ for initial candidate retrieval (fast, memory-efficient, moderate recall), followed by re-scoring with the full float32 vectors for the top candidates (slower, exact, but applied only to a small candidate set).

The float16 quantization of vector components, which reduces the per-vector memory by 2x with negligible impact on retrieval quality, is also widely used. Modern embedding models are trained with float32 precision, but the stored vectors can be safely quantized to float16 for retrieval purposes without significant recall degradation.

Vector search indexes are generally static or semi-static. While some structures (like HNSW) support incremental insertion, most achieve their best performance when built once on the full dataset. Frequent updates, such as adding or deleting documents in real time, can degrade index quality or require expensive rebuilds.

HNSW supports efficient incremental insertion: new vectors are connected into the graph during insertion without requiring a global rebuild. However, there is no efficient deletion mechanism: deleting a vector leaves "tombstoned" nodes in the graph that continue to be examined during search, wasting computation. The standard workaround is periodic full rebuilds combined with a "soft delete" approach that filters out deleted documents in post-processing.

IVF indices are more sensitive to distributional shift. If new documents are added that fall outside the existing cluster regions, the cluster assignments become suboptimal and recall degrades. Periodic retraining and rebuilding of the cluster centroids is typically necessary when the document distribution evolves significantly.

Tiered architectures address the update problem by combining a large static index (rebuilt periodically) with a small dynamic index (maintained in real time, queried with exact search due to its small size). New documents are added to the dynamic tier and periodically merged into the static tier during off-peak hours. This approach is used by large-scale production systems where both update velocity and query latency are important.

A subtle but important failure mode is metric mismatch: using a search metric that does not match the metric used during embedding model training. Many practitioners default to cosine similarity without checking whether their chosen embedding model was trained with cosine, dot product, or Euclidean distance. While the difference is often small after normalization (since normalized dot product equals cosine similarity), models trained with dot product and unconstrained vector norms may use magnitude as a meaningful signal. Using cosine similarity with such a model discards this signal.

Always read the embedding model documentation to understand the intended search metric. When in doubt, normalize your vectors: this makes the choice of metric moot and avoids metric mismatch problems regardless of how the model was trained.