Multi-Vector Retrievers (2021) › 2021: Multi-Vector Retrievers

The year 2021 marked a critical evolution in neural information retrieval, as researchers recognized that the single-vector paradigm popularized by Dense Passage Retrieval had fundamental limitations. While DPR and similar systems had demonstrated the power of dense embeddings for semantic search, they compressed entire queries and documents into single vectors, losing the fine-grained matching capabilities that had made traditional sparse methods like BM25 effective. This compression created a fundamental tradeoff: dense single-vector methods captured semantic similarity but struggled with precise term matching, exact phrase retrieval, and scenarios where individual terms mattered more than overall semantic similarity. Multi-vector retrieval emerged as a solution that combined the semantic understanding of dense methods with the precision of fine-grained matching, representing a hybrid approach that would influence retrieval systems for years to come.

By 2021, the retrieval field had settled into two distinct camps. On one side, sparse methods like BM25 remained dominant in production systems. This offered fast retrieval, interpretable scoring, and excellent performance on exact keyword matching tasks. On the other side, dense single-vector methods like DPR had shown remarkable gains on semantic similarity tasks, particularly question answering, but required significant computational resources and struggled with tasks requiring precise term matching. The gap between these approaches seemed fundamental: sparse methods operated at the term level, matching individual query terms to document terms, while dense methods compared single vectors representing entire documents. Multi-vector retrieval bridged this gap by operating at an intermediate level of granularity, encoding both queries and documents as collections of token-level vectors that could be matched flexibly.

The core innovation of multi-vector retrievers was deceptively simple: instead of encoding a query or document as a single dense vector, these systems encoded each token as its own vector, then computed relevance scores by matching query token vectors to document token vectors. This approach, pioneered in ColBERT (Contextualized Late Interaction over BERT) and refined in subsequent systems, preserved the semantic richness of dense embeddings while restoring the fine-grained matching capabilities that made sparse retrieval effective. By representing text as collections of contextualized token embeddings rather than single aggregated vectors, multi-vector systems could match individual query terms to relevant document passages while still capturing semantic relationships between terms. This hybrid capability made them particularly effective for complex queries requiring both semantic understanding and precise matching.

The impact of multi-vector retrieval extended beyond immediate performance improvements. These systems demonstrated that effective retrieval could operate at multiple levels of granularity simultaneously, matching both semantically and lexically, capturing both overall document relevance and specific passage relevance. This multi-granular matching capability proved essential for applications like open-domain question answering, where queries might contain both semantic concepts and specific entity names or technical terms that needed exact matching. Multi-vector retrievers showed that the future of retrieval lay not in choosing between sparse and dense approaches, but in combining their strengths through architectures that could operate flexibly across different levels of matching granularity.