Multi-Vector Retrievers (2021) › Limitations

Despite their advantages, multi-vector retrieval systems faced significant limitations that prevented them from fully replacing single-vector methods in all scenarios. The most fundamental constraint was computational cost. Encoding each token as a separate vector and matching query tokens to document tokens required substantially more computation than single-vector retrieval. For a query with 10 tokens and a document with 1000 tokens, a multi-vector system needed to compute 10,000 similarity scores (10 query tokens $\times$ 1000 document tokens), while a single-vector system computed just one. This quadratic scaling with document length made multi-vector retrieval computationally expensive for long documents, requiring aggressive truncation or segmentation that could affect retrieval quality.

The storage requirements for multi-vector systems were also substantial. Instead of storing one vector per document, systems needed to store many vectors per document, one for each token. This multiplied the index size, requiring significantly more storage and memory than single-vector systems. While compression techniques and approximate indexing could reduce these requirements, they came with tradeoffs in retrieval quality or computational complexity. For large-scale retrieval systems indexing millions or billions of documents, the storage overhead of multi-vector representations could be prohibitive, limiting deployment to systems with substantial computational resources.

The matching complexity in multi-vector systems also made them harder to optimize and deploy efficiently. Single-vector retrieval benefited from decades of optimization in approximate nearest neighbor search, with highly optimized libraries and hardware support. Multi-vector retrieval required more complex matching logic, making it harder to achieve the same level of optimization. The late interaction mechanism, while effective, was more computationally intensive than simple cosine similarity, and optimizing this computation across large document collections required specialized techniques and implementations that weren't always readily available.

Multi-vector systems also struggled with some of the same limitations that affected dense retrieval methods more broadly. They still required training or fine-tuning on task-specific data to achieve optimal performance, making them less immediately applicable than sparse methods like BM25 that required no training. The contextualized embeddings used by systems like ColBERT were computationally expensive to generate, requiring full forward passes through BERT for both queries and documents, which limited throughput compared to lightweight sparse methods. These computational constraints meant that multi-vector retrieval was often most effective in scenarios where retrieval quality was prioritized over speed and cost, such as in offline preprocessing or in high-stakes applications where the cost of computation was justified by the value of accurate retrieval.

The token-level matching in multi-vector systems, while enabling fine-grained matching, could also be overly sensitive to specific token choices. Queries and documents using different tokenizations or containing typos might fail to match even when they were semantically equivalent. The reliance on token-level matching meant that systems needed consistent tokenization and explicit handling of edge cases, which added complexity to implementation and deployment. While contextualized embeddings helped address some of these issues by providing semantic flexibility, the fundamental reliance on token-level operations created vulnerabilities that pure semantic matching or pure lexical matching could avoid.