RAG Architecture: Components, Timing & Design Patterns › Summary

This chapter examined the architecture of retrieval-augmented generation systems, revealing how the seemingly simple "retrieve then generate" pipeline conceals important design decisions at every stage.

The retriever component turns documents into searchable representations (sparse or dense), stores them in specialized indexes, and returns the most relevant passages at query time. The choice of retrieval method significantly affects what information reaches the generator, with sparse methods excelling at exact term matching and dense methods excelling at semantic similarity. Hybrid retrieval combining both approaches often outperforms either alone.

The generator component synthesizes answers from retrieved context using in-context learning. Prompt design matters: clear instructions about grounding responses in context, appropriate document formatting, and explicit uncertainty handling all improve output quality. The generator sees retrieved documents as additional tokens in the prompt and attends to them through the same self-attention mechanism used everywhere in the transformer.

Retrieval timing ranges from single-shot (fast, simple, suitable for most queries) to iterative (handles complex multi-step reasoning) to token-level as in RETRO (maximum relevance at maximum cost). Most production systems use single-shot retrieval with optional iteration for hard queries.

Architecture variations like Fusion-in-Decoder, end-to-end RAG/REALM, RETRO, and Self-RAG demonstrate that the basic sequential pattern is just one point in a large design space. FiD processes documents independently to scale retrieval count. End-to-end training aligns the retriever directly with generation quality. Self-RAG makes retrieval an adaptive, query-dependent decision rather than an unconditional operation.

The limitations are real: context window constraints, retriever-generator mismatch, latency accumulation, and attribution challenges each bound what RAG can reliably achieve. Understanding these boundaries helps you build systems with appropriate safeguards and realistic expectations.

The next chapter dives deep into dense retrieval: how neural networks learn to embed queries and documents into the same vector space, letting semantic matching that goes far beyond keyword overlap. This is the technology that makes modern RAG systems substantially more capable than their sparse retrieval predecessors, and understanding it is needed for building production-quality systems.