Dense Passage Retrieval & RAG (2020) › The Problem: Static Knowledge and Information Retrieval Limitations

Pre-trained language models of the late 2010s faced a fundamental constraint: their knowledge was frozen at training time. Models like BERT and GPT had been trained on large text corpora, encoding statistical patterns and factual knowledge into their millions or billions of parameters. This knowledge base, while vast, was static. Once training completed, these models could not access new information, incorporate domain-specific knowledge not present in their training data, or answer questions about events that occurred after their training cutoff date. A model trained in 2019 could not know about events in 2020, regardless of how important those events might be.

This static knowledge limitation became particularly problematic for knowledge-intensive tasks like open-domain question answering, where systems needed to answer factual questions by drawing on external knowledge sources. Researchers attempted to address this by training ever-larger models on ever-larger datasets, hoping to encode more facts into model parameters. However, this approach had fundamental limits. There were always facts not present in training data, domain-specific knowledge requiring specialized expertise, and information that changed over time. Training massive models to memorize facts was also computationally expensive and environmentally costly, raising questions about the sustainability of scaling models purely through parameter increases.

Traditional information retrieval methods, while capable of accessing external knowledge sources, faced complementary challenges when used with neural language models. Methods like BM25 relied on exact keyword matching, so differences in wording between a query and document, including paraphrases, could reduce retrieval quality. A user might ask "What causes global warming?" but relevant documents might use terms like "climate change," "greenhouse gases," or "carbon emissions" without containing the exact phrase "global warming." Sparse retrieval methods based on term frequency and document frequency could not bridge these semantic gaps effectively.

Earlier attempts to combine retrieval with neural language models had encountered significant obstacles. Some systems used retrieval as a preprocessing step, finding relevant documents and then passing them to a language model for processing. However, this two-stage approach meant that the retrieval system and language model were trained separately, with no mechanism for the language model to learn how to make use of retrieved information effectively. The retrieval system might find relevant passages, but the language model might not know how to integrate them appropriately into its generation process.

Other approaches tried fine-tuning language models on retrieval-augmented training data, but these methods required large amounts of labeled training data showing which documents were relevant for which queries. Creating these training datasets was labor-intensive and expensive, limiting the scalability of such approaches. Additionally, these systems often struggled with the precise matching needed for question answering tasks, where retrieving the exact passage containing the answer was critical for performance.

The core challenge was creating an end-to-end trainable system that could retrieve relevant information and use it for generation. Dense representations had to capture meaning beyond exact word matches so retrieval could follow semantic similarity. Retrieval also had to be integrated with generation so the language model could condition its output on retrieved context. The training procedure needed to work with limited labeled data by adapting representations from pre-trained models.