RAG Motivation: Solving Hallucinations & Knowledge Gaps › Limitations and Design Considerations

While RAG addresses many limitations of pure parametric systems, it introduces its own set of challenges that practitioners must understand before deploying RAG in production. RAG is not a silver bullet. It shifts the failure modes of a language model deployment rather than eliminating them entirely, and some of the new failure modes it introduces can be subtle and difficult to detect.

RAG systems are only as good as their retrieval. If the retriever fails to find relevant documents, the generator cannot produce correct answers, no matter how capable the underlying language model. This "garbage in, garbage out" dynamic means that retrieval quality often matters more than generation quality for the overall system's factual accuracy.

Poor retrieval can manifest in several distinct ways. The retriever might return documents that are topically related but don't contain the specific answer to the user's question. It might miss relevant documents due to vocabulary mismatch between the query and the document text: a document about "myocardial infarction" might not be retrieved for a query about "heart attacks" if the retrieval system relies purely on keyword overlap. For ambiguous queries, the retriever might fetch documents about the wrong interpretation of the question. For multi-step reasoning tasks, the retriever might need to support iterative retrieval across multiple queries, introducing compounding error.

The key insight here is that retrieval quality investment often yields higher returns than generation quality investment. Upgrading from a smaller to a larger language model might improve response fluency and reasoning depth, but if the retrieval system consistently misses relevant documents, the generated responses will remain factually deficient regardless of model size. Conversely, a highly accurate retrieval system dramatically improves factual accuracy even with a modest generation model.

This has measurable implications for system design and resource allocation. Building good retrieval requires attention to embedding model quality, indexing strategy, query preprocessing, chunk size and overlap for document segmentation, and possibly hybrid search combining dense embeddings with sparse keyword methods. We'll explore dense retrieval, hybrid search, and other techniques for improving retrieval quality in upcoming chapters.

RAG introduces additional latency compared to pure generation. The system must encode the query, search the index, retrieve documents, and incorporate them into the prompt before generation can begin. For real-time interactive applications, this overhead can noticeably impact user experience.

The latency breaks down into several components. Embedding the query typically adds 10-50 milliseconds. Vector search in the index adds 10-100 milliseconds depending on index size and whether exact or approximate search is used. Fetching document content from the document store adds variable latency depending on storage technology and network characteristics. Finally, the language model processes a longer prompt due to the retrieved context, which increases generation time proportionally to the number of retrieved tokens.

For interactive applications, the total added latency of 100-500 milliseconds may noticeably affect user experience, though modern users are accustomed to some waiting in web applications. For applications with strict real-time requirements, such as voice assistants or high-frequency trading systems, this latency may be unacceptable.

Various optimization strategies can mitigate these concerns. Caching frequent queries avoids redundant retrieval for common questions. Pre-computing embeddings for documents at indexing time avoids recomputation at query time. Using approximate nearest neighbor algorithms like HNSW or IVF reduces search latency at the cost of small accuracy reductions. Streaming generation while retrieval completes in a background thread allows users to see the beginning of the response earlier. In practice, carefully implemented RAG systems can achieve end-to-end latencies that are acceptable for most interactive applications.

As we discussed in Part XVIII on context length challenges, language models have finite context windows. Retrieved documents compete for context space with your query, system prompts, conversation history, and any other contextual information the model needs. This creates a basic tension: retrieving more documents provides more potentially relevant information but consumes more context space, potentially crowding out other important context.

This creates what practitioners call a retrieval budget problem: the system must decide how many documents to retrieve, how long each document excerpt can be, and how to handle cases where the total retrieved content would exceed the context budget. Retrieving too few documents risks missing relevant information. Retrieving too many introduces noise, because not all retrieved documents are equally relevant, and may exceed the context window entirely.

Document chunking strategies help manage this constraint. Rather than indexing entire documents as single units, practitioners break documents into smaller, focused chunks that can be retrieved and included in context more efficiently. The chunking strategy, including chunk size and overlap between adjacent chunks, significantly affects both retrieval accuracy and context utilization. We'll cover chunking strategies in detail in Part XLIV Chapter 5. The core principle is that smaller, topically coherent chunks allow more precise retrieval and more efficient use of the context budget.

The ongoing trend toward longer context windows in newer models, from 4,096 tokens in early transformers to hundreds of thousands of tokens in current frontier models, partially alleviates this constraint. As context windows grow, the tension between retrieval comprehensiveness and context budget becomes less acute. However, longer contexts also create new challenges: language models often exhibit a "lost in the middle" phenomenon where information in the middle of a long context receives less attention than information at the beginning or end. This means that simply adding more retrieved documents doesn't linearly improve accuracy; how retrieved documents are ordered and positioned within the context also matters.

Unlike pure parametric systems where knowledge is fixed at training, RAG systems require ongoing knowledge base maintenance. Documents must be added when new information becomes available, updated when existing information changes, and removed when information becomes stale or incorrect. Indexes must be rebuilt or incrementally updated to reflect changes. Quality control must ensure that documents entering the retrieval corpus are relevant and accurate, with consistent formatting.

This operational burden is often underestimated when organizations first adopt RAG. A RAG system isn't "done" at deployment. It requires continuous investment to remain useful. The question of who maintains the knowledge base, how updates are validated, how stale documents are detected and removed, and how the overall quality of the corpus is monitored are all real operational challenges that require defined processes and ongoing attention.

The maintenance challenge scales with corpus complexity. A knowledge base containing a few hundred policy documents from a single author is relatively manageable. A knowledge base containing millions of documents from diverse sources, updated by many different contributors, requires reliable governance processes to prevent quality degradation over time. Organizations that deploy RAG at scale typically develop tooling for corpus management, quality monitoring, and automated testing of retrieval accuracy.

A subtle but important failure mode occurs when the generation component contradicts or ignores the retrieved documents. This can happen when the retrieved documents contain information that conflicts with the model's strong parametric beliefs. If the model learned during training that X is true, and retrieval returns a document saying X is false (perhaps because X changed since training), the model may weight its parametric beliefs more heavily than the retrieved evidence and generate a response that ignores or disputes the correct retrieved information.

This failure mode is difficult to detect because the response may be fluent and confident, and may even include attribution to the retrieved document while subtly misrepresenting what it says. Mitigating this requires careful prompt engineering to instruct the model to treat retrieved content as authoritative, as well as evaluation procedures that specifically test cases where retrieved content contradicts the model's likely parametric beliefs.