RAG Architecture: Components, Timing & Design Patterns › Limitations and Considerations

RAG architecture decisions involve inherent trade-offs that you should understand before building systems. These limitations don't make RAG ineffective; rather, they define the boundaries within which RAG excels and help you set appropriate expectations.

Context window constraints remain a basic limitation. Even with modern models supporting 100K or more token contexts, there's a practical limit to how many documents you can retrieve. As we discussed in Part XVII on efficient attention and Part XVIII on context extension, attention costs grow with sequence length, and models can struggle to effectively use information buried deep in long contexts. Research has shown that models tend to focus on information at the beginning and end of the context, with middle sections receiving less attention. This "lost in the middle" phenomenon means retrieval quality matters enormously: retrieving the right 3 documents beats retrieving 30 marginally relevant ones. You can mitigate this by placing the most relevant document first or last in the context, by reranking retrieved documents to prioritize the highest-quality ones, or by using architectures like FiD that avoid concatenation entirely.

The retriever-generator mismatch problem arises because these components are typically trained separately. A retriever optimized for traditional information retrieval metrics may not retrieve what is most useful for generation. A passage that scores highest on BM25 might lack the specific details the generator needs, while a lower-ranked passage contains exactly the right information. The retriever knows about word overlap and semantic similarity, but it doesn't know what the generator needs to answer the specific question it's facing. End-to-end training addresses this mismatch directly, but requires substantial infrastructure. Reranking approaches use a cross-encoder (which jointly encodes query and document) to re-score retrieved documents after initial retrieval, capturing relevance signals that the bi-encoder retriever misses. We'll cover cross-encoders and reranking in later chapters.

Latency accumulates across the pipeline in compound ways. Retrieval adds round-trip time to the knowledge base (potentially remote), and longer contexts increase generation time because the attention mechanism must process more tokens. For real-time applications, this motivates careful optimization: caching frequent queries, using smaller models, or pre-computing responses for common question patterns. Each additional retrieved document adds both retrieval time (more results to rank and return) and generation time (more tokens for the model to attend over), creating compound latency effects. The relationship between $k$ and total latency is often superlinear in practice, because vector index search time grows with $k$ and generation cost grows with total context length.

Attribution and faithfulness remain active research areas that no current approach fully solves. When a RAG system generates an answer, how do you verify it came from the retrieved documents rather than from hallucinated parametric knowledge? The model might confidently state a fact that appears nowhere in the retrieved context, drawing instead from knowledge encoded in its weights during pretraining. This is particularly concerning for knowledge that changes over time: a model trained before a major news event might ignore retrieved context about that event and generate answers based on outdated weights. Current approaches include asking models to cite sources, comparing responses with and without context, or using specialized fact-verification models. None are fully reliable, making RAG a tool that reduces rather than eliminates hallucination risk. You should understand that retrieved documents provide evidence for the answer, but the system may still make errors in how it interprets or synthesizes that evidence.

Knowledge base maintenance introduces operational challenges that compound over time. Documents in your knowledge base become outdated as the world changes. Adding new documents requires re-indexing. Removing documents to comply with deletion requests requires removing them from the document store and removing their embeddings from the vector index, which some index types handle poorly. The knowledge base is a living system that requires ongoing maintenance, not a static artifact. In production, you need processes for adding new content, removing stale or incorrect content, and periodically re-evaluating whether your chunking and encoding strategies still serve your users well.

Finally, evaluation is harder than it appears. Measuring RAG quality requires both component-level evaluation (is the retriever finding the right documents?) and end-to-end evaluation (is the final answer correct?). Component metrics like recall at $k$ and mean reciprocal rank tell you about retrieval quality, but a high recall doesn't guarantee good final answers if the generator fails to synthesize correctly. End-to-end metrics like exact match, F1, or human evaluation tell you about overall quality but don't pinpoint where the system is failing. Building a complete evaluation harness that covers both components, with test cases representing the diversity of queries your users will send, is one of the most important investments you can make in a production RAG system.