Dense Passage Retrieval & RAG (2020) › The Solution: Dense Passage Retrieval and Retrieval-Augmented Generation

DPR and RAG provided complementary solutions to the knowledge access problem. DPR used dense vector representations from pre-trained language models for semantic passage retrieval. RAG built on that retriever to integrate retrieval and generation in an end-to-end trainable framework.

DPR introduced a novel approach to passage retrieval that overcame limitations of both traditional sparse retrieval methods and earlier neural retrieval approaches. The key innovation was using dense vector representations from pre-trained language models like BERT to encode both queries and passages into a shared embedding space. Rather than matching queries and documents based on exact keyword overlap, DPR learned to map semantically similar questions and passages close together in this dense vector space.

The DPR architecture used a dual-encoder approach with separate encoders for queries and passages. Both encoders were initialized with pre-trained BERT models, using the semantic representations learned during pre-training. The query encoder processed questions into dense vectors, while the passage encoder processed text passages into dense vectors of the same dimensionality. During training, the system learned to map questions and their relevant answer passages close together in this embedding space, while pushing irrelevant passages farther away.

The training process used a contrastive learning objective. Given a question, the system would retrieve multiple passages from a knowledge base, including the passage containing the answer (positive example) and other passages that did not contain the answer (negative examples). The learning objective encouraged the query and the relevant passage to have high similarity scores (measured as dot product between their embeddings), while ensuring that the query had low similarity with irrelevant passages. This contrastive approach required less labeled training data than previous neural retrieval methods because it used semantic representations from pre-trained models.

The retrieval process was straightforward: given a query, encode it into a dense vector using the query encoder, then find the passages whose embeddings had the highest dot product similarity with the query vector. This could be done efficiently using approximate nearest neighbor search techniques like FAISS, enabling retrieval from millions or billions of passages in milliseconds. DPR demonstrated that this dense retrieval approach could achieve state-of-the-art results on question answering benchmarks while requiring far less training data than previous neural retrieval methods.

RAG built on DPR's retrieval capabilities to create a unified system for knowledge-intensive generation tasks. Retrieval became part of the generation process rather than only a preprocessing step. RAG trained the language model to condition generation on retrieved documents, giving it access to information beyond its training data.

The RAG architecture consisted of two main components: a retriever and a generator. The retriever used DPR to find relevant passages from a knowledge base given an input query or prompt. The generator was a pre-trained language model (typically GPT-2 or similar) that received both the original input and the retrieved passages as context for generation. During training, both components were fine-tuned together, allowing the generator to learn how to effectively use retrieved information.

The training process involved creating a dataset of questions paired with their answers and the passages containing those answers. For each training example, RAG would retrieve candidate passages using the DPR retriever, then fine-tune the generator to produce the correct answer given both the question and the retrieved passages. This joint training allowed the system to learn which passages were most useful for which types of questions and how to integrate information from multiple retrieved passages into coherent generated text.

RAG supported two operational modes: RAG-Sequence and RAG-Token. In RAG-Sequence mode, a single retrieved passage was used to condition the entire generated sequence, appropriate when the answer could be found in a single passage. In RAG-Token mode, the system could use different retrieved passages for different tokens in the generated sequence, enabling more flexible integration of information from multiple sources. Both modes demonstrated significant improvements over baseline language models on knowledge-intensive tasks, with RAG-Token providing additional flexibility for complex questions requiring information from multiple documents.

The system's ability to cite sources represented another important advantage. Because generation was explicitly conditioned on retrieved passages, RAG could provide attribution for generated claims, showing which passages from the knowledge base informed the generated text. This capability addressed important concerns about transparency and verifiability in language model outputs, enabling users to verify claims by consulting the source passages.