RAG Architecture: Components, Timing & Design Patterns › Architecture Variations

Since the original RAG paper, researchers and practitioners have developed numerous architectural variations. Understanding these helps you choose or design the right architecture for your use case. Simpler designs may sacrifice performance or capabilities, while more capable systems cost more to build and operate. The variations are not mutually exclusive; production systems often combine elements from multiple approaches.

The standard architecture we've been describing is sometimes called "retrieve-then-read": retrieve first, then read (generate). Documents pass through a clean interface: the retriever's output becomes the generator's input. This separation creates a clear contract between components: the retriever promises to return relevant documents, and the generator promises to synthesize them into an answer.

This separation enables independent optimization of each component. You can upgrade the retriever without touching generation logic, or swap in a different LLM without modifying retrieval. This modularity also simplifies debugging: if answers are wrong, you can examine retrieved documents to determine whether the problem is in retrieval (wrong documents) or generation (right documents, wrong answer). The sequential architecture is the right starting point for most projects. Build it first, measure where quality falls short, and then consider more complex architectures only if the sequential version cannot meet your requirements.

The Fusion-in-Decoder (FiD) architecture processes each retrieved document independently through the encoder, then fuses their representations in the decoder. This approach addresses a scalability challenge with the standard concatenation approach.

Instead of concatenating all documents into a single input, which creates one long sequence that grows linearly with the number of documents, FiD keeps encoding costs constant per document by encoding each document-query pair separately. Each encoding produces a fixed-size representation, and these representations are concatenated for the decoder. The decoder then attends over all document representations simultaneously when generating the answer.

This approach scales better with the number of retrieved documents. Standard concatenation creates a sequence of length $n \times k$ where $n$ is average document length and $k$ is the number of documents, and attention cost grows quadratically with sequence length. FiD keeps each encoder pass at length $n$, with only the decoder needing to handle the combined representations. The decoder's cross-attention mechanism attends over the concatenated encoder outputs, effectively fusing information from all documents. This allows FiD to process many more documents than would be practical with simple concatenation, often improving quality on tasks that require synthesizing information from many sources.

The original FiD paper demonstrated state-of-the-art performance on open-domain question answering benchmarks by retrieving and fusing up to 100 documents per query. At that scale, concatenation would create sequences of tens of thousands of tokens, making generation prohibitively expensive, while FiD kept costs manageable by encoding documents independently. The trade-off is that FiD requires an encoder-decoder architecture (like T5), which makes it less applicable to pure decoder models like LLaMA or GPT, which are increasingly common in modern RAG systems.

The original RAG paper from Facebook AI and the REALM paper from Google both proposed end-to-end trainable retrieval. Rather than using a frozen retriever, these architectures backpropagate gradients through the retrieval step. This means the retriever can learn which documents are similar to the query and which documents are useful for generating correct answers.

The key insight is treating retrieval as a latent variable. In this formulation, we don't commit to using a single retrieved document. Instead, we consider multiple retrieved documents and weight their contributions by how relevant the retriever believes them to be. Given query $q$ and a set of retrieved documents $\{d_1, \dots, d_n\}$, the probability of generating answer $a$ is computed by marginalizing over the documents:

$$P(a|q) = \sum_{i=1}^{n} P(a|q, d_i) \cdot P(d_i|q)$$

where:

• $a$: the generated answer we want to produce
• $q$: the input query from the user
• $d_i$: the $i$-th retrieved document from the knowledge base
• $n$: the total number of retrieved documents under consideration
• $P(a|q, d_i)$: the probability of the generator creating answer $a$ given query $q$ and document $d_i$, measuring how well this specific document helps generate the correct answer
• $P(d_i|q)$: the probability that the retriever assigns to document $d_i$ being relevant to query $q$, derived from the similarity score between the query embedding and the document embedding

This formula computes a weighted mixture of generation probabilities, where each document's contribution to the final probability is proportional to how relevant the retriever believes it to be. Think of it as holding a committee vote: each document votes on the answer, and its vote carries weight proportional to its relevance score. If document $d_1$ is highly relevant ($P(d_1|q)$ is large) and strongly supports the correct answer ($P(a|q, d_1)$ is large), it dominates the sum. Documents that are irrelevant or that point toward wrong answers receive low weights and contribute little.

The core advantage of this formulation is differentiability. Because both terms in the product are differentiable functions of model parameters, gradients flow from the generation loss back through the generation probability, through the retriever weights, to the document embeddings. If the generator consistently produces wrong answers when using certain documents, the retriever learns to down-weight those documents for similar queries. The retriever and generator co-adapt: the retriever learns to retrieve what helps the generator, and the generator learns to use whatever the retriever provides.

Training end-to-end is expensive because updating document embeddings requires re-indexing, which can take hours for large knowledge bases. This creates a practical challenge: if you update the document encoder at every gradient step, you need to re-embed and re-index all documents continuously, which is impractical. The original RAG paper uses asynchronous index updates, refreshing document embeddings periodically (e.g., every few thousand gradient steps) rather than after every update. However, the resulting retrievers are specifically optimized for the generation task, often outperforming generic retrievers by substantial margins on knowledge-intensive benchmarks. The trade-off is infrastructure complexity and training cost versus retrieval quality.

The RETRO (Retrieval-Enhanced Transformer) architecture from DeepMind takes a different approach, building retrieval into the transformer architecture itself. Rather than treating retrieval as a preprocessing step, RETRO makes retrieval an integral part of the model's forward pass.

RETRO divides the input into chunks and retrieves neighbors for each chunk. These neighbors are documents from the knowledge base that are similar to the current chunk being processed. Retrieved neighbors are processed by a separate encoder, and their representations are integrated through chunked cross-attention layers interleaved with standard self-attention. This means the model alternates between attending to its own representations (self-attention over the input and generated text) and attending to retrieved content (cross-attention over the encoder outputs for retrieved passages). Every chunk of generated text has access to the most relevant passages from the retrieval database, dynamically updated as generation proceeds.

This tight integration allows RETRO to scale retrieval to trillions of tokens while keeping the main model relatively small. The model can effectively "look up" information during generation rather than storing all knowledge in its parameters. A 7B parameter RETRO model can match the performance of a 25B model without retrieval by using a massive retrieval database. This represents a major change in how we think about model capacity: some knowledge lives in parameters (where it's always available but may be outdated), while other knowledge lives in an external database (where it can be updated without retraining) accessed through retrieval. The implication is that the right strategy for scaling might be to scale the retrieval database rather than the model parameters, at least for knowledge-intensive tasks.

Most RAG systems retrieve for every query, but retrieval isn't always necessary or beneficial. Simple queries like "What is 2+2?" don't benefit from retrieval, and retrieving irrelevant documents can hurt performance by introducing noise and confusing the generator. Self-RAG trains models to decide when to retrieve, what to retrieve, and how to use retrieved content, adapting retrieval behavior to the specific query at hand.

The Self-RAG model generates special control tokens during its forward pass, inserting them at appropriate points to indicate its decisions about retrieval and document relevance:

• [Retrieve]: Signals whether retrieval is needed for this query or generation segment
• [IsREL]: Assesses whether a retrieved passage is relevant to the current generation context
• [IsSUP]: Evaluates whether a passage supports the claim being generated
• [IsUSE]: Determines whether to use a passage in the final generated response

These reflection tokens enable the model to critique its own retrieval usage in real time, filtering irrelevant documents and identifying when it should rely on parametric knowledge instead of retrieved context. The model has been trained through a combination of supervised learning (to generate appropriate reflection tokens) and reinforcement learning (to optimize the quality of final answers), so its retrieval decisions are grounded in observed answer quality.

Self-RAG is more adaptive than systems that blindly retrieve for every query. It achieves competitive quality with selective retrieval, reducing both latency and cost for queries that don't need external information. The training approach also produces models that express more calibrated uncertainty: they retrieve more when they're less sure about the answer and rely on parametric knowledge when they're confident, rather than always deferring to the retriever. This behavior is closer to how a knowledgeable human expert would work: drawing on memory for familiar facts while looking things up for less familiar topics.

Rather than choosing between sparse and dense retrieval, many production systems use both and combine their results. The intuition is simple: sparse retrieval is excellent at finding documents with exact term matches (useful when queries use precise technical terminology), while dense retrieval is excellent at finding semantically related documents (useful when queries use everyday language about technical topics). Combining both captures the strengths of each.

The most common combination strategy is Reciprocal Rank Fusion (RRF). Given two ranked lists of documents, one from sparse retrieval and one from dense retrieval, RRF assigns each document a score based on its rank in each list:

$$\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)}$$

where:

• $d$: the document being scored
• $R$: the set of retrieval methods (e.g., sparse and dense)
• $\text{rank}_r(d)$: the rank of document $d$ in the results from retrieval method $r$ (lower rank means more relevant)
• $k$: a constant (typically 60) that dampens the influence of very high-ranked documents and prevents any single result from dominating

The key insight behind RRF is that rank is more reliable than score. BM25 scores and cosine similarities live on different scales and cannot be directly compared, but if a document ranks first in both the sparse and dense results, that convergent signal strongly suggests relevance. The constant $k = 60$ was found empirically to work well across many retrieval tasks by reducing the impact of randomly high rankings. Documents that appear in the top results of both systems receive the highest RRF scores, with the score declining as rank falls.

RRF is simple to implement, requires no model training, and consistently outperforms either retriever alone on most benchmarks. More sophisticated fusion approaches learn weights for each retrieval signal, but RRF is an excellent default that works well across diverse domains.