Vector Similarity Search: Metrics & Approximate Methods › Summary

This chapter established the foundations of vector similarity search, the core retrieval operation in any RAG pipeline.

Distance metrics define what "similar" means in vector space. Cosine similarity (direction-based) is the default for text embeddings, while dot product is equivalent for normalized vectors and Euclidean distance captures position-based proximity. Always match your search metric to your embedding model's training objective. For normalized vectors, all three metrics induce identical rankings, which is why pre-normalizing at index time is standard practice.

Exact search computes distances against every vector in the collection, guaranteeing perfect recall at $O(n \cdot d)$ cost per query. It is practical for small collections (under roughly 1 million vectors on CPU, or more with GPU batching) but does not scale. Its main role is as a ground-truth generator for evaluating ANN recall.

Approximate Nearest Neighbor (ANN) algorithms achieve sublinear query time by sacrificing a small, controllable amount of recall. The key families are tree-based (efficient in low dimensions, but limited in high dimensions), hash-based (LSH, simple and parallelizable but memory-intensive), graph-based (HNSW, currently dominant due to excellent recall-speed balance), partition-based (IVF, good for very large collections), and compression-based (Product Quantization, for memory-constrained settings).

FAISS is the most widely used similarity search library. This provides implementations of all major index types with both CPU and GPU backends. The recall-vs-speed trade-off curves we generated show how parameter tuning lets you find the right operating point for your application. Libraries like HNSWlib, Annoy, and ScaNN offer specialized alternatives, while vector databases like Qdrant and Milvus add persistence and operational features for production deployments.

The limitations of vector search (semantic gap, curse of dimensionality, memory constraints, update costs) motivate the full RAG pipeline architecture, where approximate retrieval feeds into reranking and other refinement stages that compensate for the inherent approximations of embedding-based search.

With the conceptual framework and metrics in place, the next three chapters examine the specific index structures that make large-scale ANN search possible: HNSW graphs, IVF partitioning, and Product Quantization.