RAG Architecture: Components, Timing & Design Patterns › RAG Architecture and Design Patterns

The previous chapter established why retrieval-augmented generation matters: it grounds language models in external knowledge, reducing hallucinations and letting access to information beyond the training cutoff. But understanding the motivation is only half the story. To build effective RAG systems, you need to understand how the components fit together and the design decisions that shape system behavior. A RAG system that retrieves poorly will produce wrong answers even if the generator is perfect. Conversely, a weak generator will fail to synthesize correct answers even when the retriever finds exactly the right documents. The interaction between these components, and the way you orchestrate that interaction, determines whether your system succeeds or fails in practice.

Think of a RAG system as a research assistant who has access to a library. When you ask a question, the assistant first searches the library's catalog and retrieves the most relevant books and articles. Then, with those materials spread on the desk, the assistant reads through them and composes an answer. The quality of the assistant's response depends on two things: how good the librarian is at finding relevant materials (retrieval), and how skilled the assistant is at reading comprehension and synthesis (generation). A brilliant researcher who retrieves the wrong books will give you a wrong answer. An excellent librarian who supplies perfect sources to an incompetent reader will also fail. Both components must work, and they must work together.

A RAG system consists of two primary components: a retriever that finds relevant documents from a knowledge base, and a generator (typically a large language model) that synthesizes an answer from those documents. These components interact through a carefully designed interface, and the timing and manner of their interaction significantly affects system performance. The retriever acts as a search engine, filtering millions of documents to identify the few most relevant ones. The generator then reads these documents and synthesizes a response that addresses your question. But this description understates the complexity: every decision about how to encode documents, what index structure to use, how many documents to retrieve, how to format the context, and when retrieval occurs can measurably shift final system quality.

This chapter examines each component in detail, explores when retrieval occurs relative to generation, and surveys the major architectural variations that have emerged since the original RAG paper. We also build a complete working pipeline so you can see these concepts in code. By the end, you'll understand the design space well enough to make informed decisions when building your own systems, and you'll know which trade-offs matter most for different use cases.