Dense Passage Retrieval & RAG (2020) › 2020: Dense Passage Retrieval and Retrieval-Augmented Generation

In 2020, researchers at Facebook AI Research (now Meta AI) introduced two closely related innovations that would fundamentally change how language models interact with knowledge: Dense Passage Retrieval (DPR) and Retrieval-Augmented Generation (RAG). These developments addressed a critical limitation of pre-trained language models: their inability to access and incorporate external knowledge beyond what was encoded in their parameters during training. While models like BERT and GPT had achieved remarkable performance on language understanding and generation tasks, they were constrained by their static knowledge base, unable to access up-to-date information, domain-specific knowledge, or factual details not present in their training data. DPR and RAG showed that combining dense vector retrieval with neural language models could create systems capable of answering questions using external knowledge sources, generating more accurate and factual text, and adapting to new information without retraining.

By 2020, the field had seen substantial progress in both information retrieval and language modeling, but these areas had developed largely independently. Information retrieval systems used sparse keyword-based methods like BM25 or neural approaches with learned query-document representations to find relevant documents from large collections. Language models like BERT and GPT had become powerful tools for understanding and generating text, but they operated as closed systems with knowledge frozen at training time. The question of how to combine these capabilities had been explored in limited ways: some systems used retrieval as a preprocessing step to find relevant context before generation, while others fine-tuned models on retrieval-augmented training data. However, no approach had successfully integrated dense semantic retrieval with neural language generation in an end-to-end trainable framework that could learn to retrieve relevant information and use it effectively for generation tasks.

DPR and RAG influenced systems beyond question answering. They established retrieval-augmented generation as a method for building knowledge-aware language systems by integrating dense vector search directly with neural language models. Later systems applied this method to chatbots, multiple knowledge sources, and retrieval during a conversation. The work offered an alternative to training ever-larger models to memorize more facts: retrieve external evidence when a task requires it.