RAG Architecture: Components, Timing & Design Patterns › Retrieval Timing

A important architectural decision is when retrieval occurs relative to generation. This timing choice affects latency, complexity, and the types of queries the system can handle effectively. Different timing strategies suit different use cases, and the choice determines the basic capabilities of the system rather than merely optimizing performance.

Think of retrieval timing as the difference between a student who reads all relevant materials before writing an essay (single-shot), one who writes sections and periodically stops to look up needed facts (iterative), and one who has a photographic memory aid that supplies relevant passages for each word they write (token-level). Each approach has different strengths: the first is simpler and faster, the second handles complex multi-part questions better, and the third maintains maximum relevance at maximum cost.

The simplest approach retrieves documents once, before generation begins. The query goes to the retriever, documents come back, and generation proceeds with that fixed context. The flow is linear: your query reaches the retriever, which searches the index and returns relevant documents, then those documents and the original query combine into a prompt that the generator processes to produce the final response.

Single-shot retrieval works well when:

• The query clearly specifies what information is needed
• Retrieved documents are likely to contain complete answers
• Low latency is necessary (one retrieval round-trip)
• The query does not require multi-hop reasoning across different topics

Most production RAG systems use single-shot retrieval because it handles straightforward queries quickly with a simple architecture. The design is easy to reason about and debug, which also makes optimization more direct. When something goes wrong, you can examine the retrieved documents to determine whether the problem was in retrieval (wrong documents retrieved) or in generation (right documents, wrong answer). This diagnostic clarity is a significant practical advantage.

The limitation of single-shot retrieval becomes apparent with complex queries. Consider "How did the policy changes in 2019 affect the 2020 research outcomes described in the Smith et al. study?" This query requires knowing what the 2019 policy changes were, what research outcomes the study described, and how the former affected the latter. A single retrieval might return some of the relevant documents, but might miss the policy context entirely, or return the study abstract without the relevant methodology section. Single-shot retrieval cannot adapt to what it finds.

Complex queries sometimes require multiple retrieval steps. The model might need to first retrieve background information, then use that to formulate more specific sub-queries. In iterative retrieval, the process begins with an initial retrieval based on your query. Partial generation may then reveal a gap in the available information, prompting the system to formulate a new, more specific query and retrieve again. The final generation step synthesizes all retrieved content into a complete response.

Consider the query: "Compare the economic impact of the 2008 and 2020 recessions." A single retrieval might not return sufficiently comparable information about both events. The query contains two distinct information needs, and the knowledge base might organize information about each recession separately. Iterative retrieval allows the system to retrieve documents about the 2008 recession first, then retrieve documents about the 2020 recession, and finally synthesize a comparison using both sets of documents.

The trade-off is latency and complexity. Each retrieval adds round-trip time and requires logic to determine when additional retrieval is needed. The system must decide how to decompose complex queries and when it has gathered sufficient information to generate a final answer. In production systems, iterative retrieval often involves a "planning" step where the model first identifies the sub-questions that need answering, then retrieves for each one, and finally synthesizes the collected information. This is closely related to agent-based systems where a language model orchestrates multiple tool calls, with the retriever being one of the available tools.

Iterative retrieval can also implement multi-hop reasoning, where answering one question provides context needed to answer the next. For example, to answer "Who founded the company that makes the most popular Python web framework?", a system might first retrieve "What is the most popular Python web framework?" (answer: Django), then retrieve "Who founded Django?", letting a two-hop answer that neither single query alone could provide. Multi-hop reasoning is particularly important for knowledge-intensive tasks where the answer requires combining information from multiple distinct sources.

At the opposite extreme from single-shot, some architectures retrieve new documents for each token generated. The RETRO (Retrieval-Enhanced Transformer) architecture from DeepMind interleaves retrieval throughout generation, attending to freshly retrieved passages at regular intervals. This approach ensures that the retrieved context remains relevant even as generation shifts to new topics or sub-questions.

RETRO divides the input into chunks of a fixed size (typically 64 tokens) and retrieves nearest-neighbor passages from a massive database for each chunk. These retrieved passages are processed by a separate encoder and integrated into the main model through cross-attention layers. The effect is that every portion of the generation process has access to relevant passages from the retrieval database, dynamically updated as the generation proceeds. A passage relevant to the first part of a long answer might not be relevant to the third part, and RETRO can adapt by retrieving different passages for each segment.

This approach can maintain relevance as generation shifts topics, but the computational overhead is substantial. Each retrieval operation adds latency, and performing thousands of retrievals (one per chunk of generation) quickly becomes impractical for interactive applications. Token-level retrieval is primarily a research technique rather than a production pattern, though its insights have influenced more practical architectures. The key insight that retrieval should remain dynamically relevant throughout generation, rather than being fixed at query time, has influenced designs like kNN-LM and more recent memory-augmented approaches.

An orthogonal timing dimension is when document processing occurs. This choice affects the trade-off between preparation time (when documents are added) and response time (when queries arrive).

Index-time processing pre-computes everything possible: chunking, embedding, and indexing happen once when documents are added to the knowledge base. Query-time only computes query encoding and index lookup. This approach minimizes latency for users but requires re-processing documents whenever the chunking strategy or embedding model changes. If you want to upgrade from a 384-dimensional embedding model to a better 1536-dimensional model, you need to re-embed every document in your knowledge base, which can take hours or days for large corpora.

Query-time processing delays some computation until a query arrives. For example, the system might store raw documents and compute embeddings using a query-dependent prompt that incorporates information about the specific query or user context. This enables more sophisticated relevance scoring at the cost of latency. Some hybrid approaches pre-compute base embeddings at index time but compute additional query-specific features at query time, combining the efficiency of pre-computation with the flexibility of dynamic scoring.

Most systems favor aggressive index-time processing to minimize query latency, accepting the overhead of re-indexing when configurations change. This matches the typical read/write pattern: documents are added infrequently, but queries arrive continuously. Optimizing for query latency by accepting higher index-building time is usually the right trade-off.