RAG Architecture: Components, Timing & Design Patterns › The Generator Component

The generator takes the retrieved documents and the original query, then produces a natural language response. In modern RAG systems, this is almost always a large language model, either an API-based model like GPT-4 or an open-weight model like LLaMA. The generator's role extends beyond simple information extraction: it must understand the query's intent, identify relevant portions of the retrieved documents, resolve any conflicts between sources, and synthesize a coherent, well-structured response. This is fundamentally a reading comprehension and synthesis task, and the generator's effectiveness depends heavily on both its underlying capabilities and how you present the context.

Think of the generator as a lawyer who has been handed a stack of case files and asked to write a brief. The lawyer doesn't just quote the files verbatim; they read critically, synthesize the relevant points, identify the strongest evidence, and construct a coherent argument. When sources conflict, they weigh the evidence. When sources are silent on a key point, a good lawyer acknowledges the gap rather than fabricating. The generator faces the same challenge: read the retrieved documents, identify what's relevant to the query, synthesize a response, and know when to say "I don't have enough information."

The quality of the generator's output depends on three interacting factors: the quality of the retrieved context (garbage in, garbage out), the quality of the prompt (which shapes what the model attends to and how it responds), and the intrinsic capability of the underlying model (which determines what complex synthesis and reasoning are possible). We can control all three, and improving any one of them generally improves the final answer.

The most straightforward approach to context integration is prompt concatenation: retrieved documents are inserted into the prompt before the query, giving the model access to relevant information through its standard attention mechanism. This approach requires no architectural modifications to the language model; we simply provide additional context that the model can reference during generation.

A typical RAG prompt structure looks like:

The generator reads this concatenated input and generates a response, attending to both the query and the retrieved documents. This uses the in-context learning capabilities we discussed in Part XXVIII: the model learns to extract and synthesize information from the provided context without any weight updates. The model has been trained on countless examples of reading passages and answering questions, so it naturally applies these learned behaviors to the RAG setting. The point is that prompt concatenation turns the generation task into something the model already knows how to do well. You are not asking the model to learn a new skill; you are giving the raw material and asking it to apply an existing skill.

The context window is both the enabler and the constraint here. With a 128k or 200k token context window, you can retrieve many documents and present them all simultaneously. The model can attend to all of them in a single forward pass, drawing connections across documents that iterative approaches would miss. But larger contexts come with costs: latency increases because the model must process more tokens, and attention costs scale quadratically with sequence length (unless you use efficient attention variants). There is also evidence that models do not use all positions in a long context equally well, with positions in the middle of a very long context receiving systematically less attention than positions at the beginning and end.

From the model's perspective, retrieved documents are simply additional tokens in the input sequence. There is no special mechanism distinguishing context from query; both are processed uniformly by the same transformer layers. The self-attention mechanism, which we covered extensively in Part XIII, allows every generated token to attend to every token in the context. This means the model can identify which parts of which documents are relevant to the query by computing high attention weights for informative passages and low weights for irrelevant ones. It can synthesize information across multiple documents, combining facts from different sources into a unified answer. When sources conflict or provide complementary information, the model can implicitly judge which source is more authoritative or relevant based on patterns learned during training.

The attention patterns often reveal which documents influenced the response. Tokens in the generated answer attend strongly to the specific passages they're drawing from. This creates an implicit citation mechanism. Researchers have exploited this property to build attribution systems that show which retrieved passages contributed to each part of the generated response. While not a formal guarantee of faithfulness, these attention patterns provide valuable interpretability. Tools like LLM-Grader and various attribution frameworks attempt to use these signals to produce document-level or span-level citations, helping users verify that claims in the generated answer are grounded in the retrieved context rather than hallucinated.

The generator's ability to use retrieved context depends on how well it was trained to do so. Base language models trained only on next-token prediction may not reliably distinguish context-based answers from parametric answers. Models fine-tuned specifically on reading comprehension tasks or instruction-following tend to ground their responses in the provided context more reliably. This is one reason why using a model fine-tuned for question answering or instruction following often improves RAG quality over using a raw language model, even if the base models have similar overall quality.

The exact prompt template significantly affects generation quality. Small changes in wording can substantially alter the model's behavior, determining whether it faithfully uses the context, hallucinates confidently, or appropriately expresses uncertainty. Prompt engineering for RAG is a non-trivial skill, and the right template often depends on the specific model you're using.

Several considerations shape effective RAG prompts:

• Instruction clarity: Explicitly telling the model to base its answer on the provided context reduces hallucination. Phrases like "Answer based only on the documents above" or "If the information isn't in the context, say you don't know" provide clear behavioral guidance. Without such instructions, many models will blend retrieved context with their parametric knowledge, which can introduce outdated or incorrect information.
• Document ordering: Models sometimes exhibit position bias, attending more to documents at the beginning or end of the context. Important documents might be placed in these privileged positions, or the order might be randomized to reduce systematic bias. Some research suggests ordering by decreasing relevance score (most relevant first) helps, while other research suggests the opposite. The safest approach is to experiment on your specific domain and model.
• Attribution requests: Asking the model to cite which documents it used can improve traceability. Requests like "Reference the document numbers in your answer" encourage the model to explicitly connect claims to sources, which makes it easier to verify generated content.
• Uncertainty expression: Instructing the model to say "I don't know" when context is insufficient prevents confident hallucinations. Without this instruction, models often generate plausible-sounding answers even when they lack the information to do so correctly. An explicit instruction to express uncertainty shifts the model's behavior from confabulation toward appropriate epistemic humility.
• Format specification: Telling the model how to structure its response (bullet points, numbered steps, prose paragraphs) improves usability and consistency. Consistent output formats also make downstream parsing more reliable if your application processes the generated text programmatically.

This prompt template makes the task explicit: use the documents, cite uncertainty when appropriate. The numbered document format enables the model to reference specific sources in its response, improving traceability and helping you verify the generated information. Notice how the instruction at the beginning explicitly constrains the model to the provided context, which is important for faithfulness.

When using chat-based APIs (GPT-4, Claude, etc.), you have additional choices about where to place context. Retrieved documents can go in the system prompt (which persists across turns in a multi-turn conversation) or in the user message (which is specific to each turn). System prompt placement is useful when you have stable context documents that should inform all responses, like a company's product documentation. User message placement is better for dynamic context that changes per query, which is the typical RAG case. Some practitioners place general instructions in the system prompt ("You are a helpful assistant that answers questions based on provided context") and the actual retrieved documents in the user message alongside the query, striking a balance between reusability and freshness.

The placement decision also affects caching behavior. Some inference providers cache system prompts, so placing frequently-used context there can reduce latency and cost. Prefix caching allows the provider to reuse the KV-cache computation from a shared prefix, avoiding redundant processing. If your application has a fixed document set that many users query, system-prompt placement with prefix caching can significantly reduce latency.