RAG Prompt Engineering: Context Placement

Retrieval-Augmented Generation grounds large language models in external knowledge, but the retrieved context is only useful if the model uses it. As we discussed in the earlier chapters on embedding-based retrieval, we can train models to retrieve relevant documents with high precision, and as we explored in the chapter on reranking, we can refine those results for maximum relevance. Yet none of this pipeline work matters if the prompt construction fails to present that context effectively. A perfectly retrieved document placed in the wrong position, wrapped in ambiguous formatting, or accompanied by vague citation instructions might as well not exist.

Think of RAG prompt engineering as the final mile problem of information retrieval. Your retrieval system has done the hard work of finding the needle in the haystack; prompt engineering is how you hand that needle to the model without it getting lost in the pile again. The model is a sophisticated reader, but one with well-documented quirks: it pays more attention to the beginning and end of what you give it, it can be confused by inconsistent formatting, and it will sometimes ignore explicit instructions when its parametric memory pulls strongly in a different direction. Understanding these quirks, and designing prompts that account for them, is what separates reliable RAG systems from ones that hallucinate despite having the right documents on hand.

RAG prompt engineering combines traditional prompt design with context window management. The model must perform a task while synthesizing information from potentially dozens of retrieved passages, maintaining fidelity to sources, handling irrelevant or contradictory information, and staying within token limits. Prompt structure, context placement, citation formatting, and instructions determine whether the response is grounded and accurate or hallucinated. Every decision you make in the prompt template, from how you label documents to how you phrase the instruction to cite sources, shapes the model's output in measurable ways.

This chapter covers the key engineering decisions in RAG prompt construction: where to place retrieved context in the prompt, how to order multiple documents, what citation formats work best, how to truncate long documents while preserving relevance, and how to write instructions that reliably guide model behavior. We treat these as engineering problems with measurable tradeoffs, not stylistic choices, because your production system's accuracy depends on getting them right. We will build toward a complete prompt-building implementation that you can adapt for your own RAG pipelines, and we will examine the failure modes that haunt even well-designed systems.

When you retrieve documents for RAG, you face an immediate architectural question: where do these documents go in the prompt? Unlike the fixed instructions of a system prompt or the specific query from a user, retrieved context is variable in length, potentially noisy, and important to the final output. The answer is not as simple as "put them before the question" because large language models do not process all positions in their context window with equal attention. Position measurably affects whether the model uses the information you provide.

Think of the context window as a theater where the seats at the front and back are premium. Information placed in those positions gets the most attention from the audience (the model), while information buried in the middle rows often goes unnoticed despite being present. The challenge for RAG systems is that you typically retrieve between 5 and 20 documents, some of which are only marginally relevant, and you need to pack them into a finite window in a way that maximizes the chance the truly important passages get read carefully.

The placement decision interacts with everything else in your prompt. A strong system instruction placed before the context competes for primacy with the documents themselves. A long list of retrieved passages placed immediately before the user query exploits recency bias but risks truncation if the concatenated context is too long. Understanding the mechanics of position bias, and then making deliberate choices about how to fight or exploit it, is the first major skill of RAG prompt engineering.

Language models exhibit position bias when processing long contexts. Studies of large language models reveal a U-shaped performance curve: models excel at using information at the very beginning and very end of their context window, but struggle with information buried in the middle. This "lost in the middle" problem means that your most relevant document, if placed poorly, might be effectively ignored.

The bias stems from how attention mechanisms process long sequences. As we explored in the chapter on attention complexity, attention costs grow quadratically with sequence length, and in practice, models employ various approximations and soft attention decay that favor proximal tokens. Even with techniques like rotary position embeddings, the practical reality is that not all positions in a long context receive equal processing fidelity. The model's internal representations are shaped by what is in the context and where each piece of information appears.

To understand this phenomenon mathematically, consider the attention mechanism as a function of position. In a transformer architecture, the attention weight between a query at position $i$ and a key at position $j$ typically incorporates positional information through either absolute embeddings or relative position biases. As the distance $|i - j|$ increases, the effective attention capacity diminishes due to several interacting factors.

The first factor is softmax saturation. The attention scores undergo softmax normalization across all positions. In long sequences, the probability mass dilutes across many tokens, reducing the gradient flow to distant positions. Each token in a short context receives a meaningful share of the attention budget; each token in a long context competes with hundreds of others. The second factor is positional embedding degradation. Whether using sinusoidal encodings or learned embeddings, the distinctiveness of positional representations tends to blur for middle positions in very long contexts, creating an information bottleneck where the model struggles to distinguish "token at position 512" from "token at position 513" in a meaningful way. The third factor is context window pressure. Modern large language models often employ attention approximations, such as sparse attention patterns or sliding window attention, which explicitly limit the receptive field of each token to its local neighborhood.

The U-shaped performance curve emerges from the interaction between these architectural constraints and the sequential processing biases inherent in autoregressive generation. Information at the beginning of the context establishes the foundational semantic framework for the model's processing, while information at the end benefits from recency effects in the attention mechanism and proximity to the query tokens. Middle positions suffer from both diminished attention weights and cumulative context drift, where the model's internal state has been transformed by processing preceding tokens, potentially obscuring the relevance of middle content.

The key insight is that the U-shape is not uniform across all models or all context lengths. Smaller models with shorter training context windows show steeper drops in the middle. Newer models trained with techniques specifically designed to improve long-context utilization show shallower drops, but the effect persists even in the most advanced models. You should design your RAG prompt as if the middle will be partially ignored, because even if your specific model handles it better than average, the cost of that assumption being wrong is a hallucinated or grounded-only-on-edge-documents response.

For RAG systems, this creates a strategic dilemma. Beginning placement guarantees the model sees the context early, but may be overshadowed by the final instruction or query when the model generates its response. End placement, placing context just before the query, takes advantage of recency bias but risks truncation if the context is too long. Interleaved placement, weaving context throughout the conversation history, complicates attribution and is rarely worth the implementation complexity. Most production systems use a form of strategic boundary placement that we will explore in the next section.

When you have multiple retrieved passages, their order matters beyond simple position bias. The goal is to arrange documents so that the most important ones occupy the high-attention positions at the beginning and end of the context block, while accepting that documents in the middle will receive less careful processing.

Consider these strategies and their tradeoffs:

• Relevance-based ordering (Sandwich pattern): Places the highest-scoring document first and the second-highest document last, with lower-confidence documents in the middle. This exploits both primacy and recency effects, sacrificing the marginal documents to ensure the best ones are processed fully. This is the default recommended strategy for most RAG applications.

• Chronological ordering: Arranges documents by publication date or version, useful when recency matters more than retrieval score. A query about current regulations or recent events may be better served by a slightly less semantically similar but more recent document than by a highly similar but outdated one.

• Cluster-based grouping: Groups related documents together, reducing the cognitive load on the model to connect dispersed information. This works particularly well when you have retrieved documents that collectively address different facets of a complex question, and you want the model to process each facet as a coherent unit before moving to the next.

• Authority-based ordering: For domains with hierarchical information authority (legal, medical, regulatory), placing authoritative sources first and supplementary sources later mirrors how human experts read these materials. A Supreme Court ruling before a law review article is not just polite convention; it signals to the model which source to treat as controlling.

The sandwich pattern deserves a closer look because it is the most widely adopted strategy in production systems. The intuition is simple: if you have $n$ documents ranked by relevance $s_1 \geq s_2 \geq \ldots \geq s_n$, you place document 1 at position 1 (primacy), document 2 at position $n$ (recency), and fill the middle with documents 3 through $n-1$ in any order. The two most important documents are guaranteed to land in the high-attention zones, and the less important documents absorb the attention deficit of the middle positions. This is a heuristic rather than a formally optimal strategy, but it is well-motivated by the empirical evidence and easy to implement.

The sandwich pattern assumes your relevance ranking is reliable. If your retrieval system returns documents with scores clustered tightly together, say all between 0.75 and 0.80, then the distinction between first and second place is more noise than signal, and you might prefer a different strategy such as random ordering within a relevance tier, or clustering by subtopic so that the model processes related passages together. The best ordering strategy depends on your retrieval quality as much as on model architecture.

A critical requirement for production RAG systems is verifiability. You need to know which retrieved passage supported which claim in the generated response. Without citations, you cannot audit the model's outputs, you cannot build trust with users, and you cannot diagnose failures. This requirement shapes how you instruct the model to cite and how you format the retrieved documents in the first place. The citation format you choose in the context portion of the prompt must match the citation format you request in the output portion, and both must be simple enough that the model can reliably execute them under generation pressure.

Think of the citation system as a contract between you (the prompt author) and the model. You agree to label your sources consistently, and in return you ask the model to reference those labels when it uses the information. The more clearly and consistently you fulfill your side of the contract, the more reliably the model will fulfill its side. Ambiguous labeling, inconsistent formatting across documents, or vague citation instructions break this contract and lead to the hallucinated or omitted citations that plague naive RAG implementations.

The choice of citation format also affects the readability of the output for your end users. A system that produces academic-style footnote references serves a different audience than one that produces inline links or numbered superscripts. It is worth deciding what the output should look like for users before working backward to design the prompt that produces it, rather than starting with whatever is easiest to implement and adjusting later.

The most common approach embeds citation markers directly in the context, assigning each document a short identifier:

You then instruct the model to cite sources using these markers:

This creates a clear mapping between claims and sources, but introduces several challenges. The model must maintain the mapping between bracketed numbers and content across thousands of tokens, and it must learn to use the specific citation syntax you provide. When you have twenty documents, maintaining that mapping under the distributional pressure of language generation becomes error-prone. The model may cite [Document 7] when it means [Document 17], or correctly identify that the claim comes from a specific passage but misremember which number was assigned to it.

A less position-sensitive variant uses content-derived identifiers rather than sequential numbers. If you label your documents with meaningful IDs like [hathitrust_2014] or [campbell_1994], the model can draw on its parametric knowledge of those identifiers to make the association, rather than relying purely on positional counting in the prompt. This works especially well in specialized domains where the model has seen many of these documents during pre-training.

Systems can also use structured markup that clearly delineates document boundaries:

XML-style formatting provides explicit structure that helps the model distinguish document content from its metadata and boundaries. The closing tags create clear delimiters that are less ambiguous than simple newlines or brackets. Modern language models have processed vast amounts of XML during pre-training and handle this structure naturally, often without needing any special instructions about how to interpret the tags.

The metadata attributes within the opening tag serve a dual purpose. They provide the model with provenance information (title, date, source type) that it can use when reasoning about authority and recency, and they give you, the developer, a way to trace exactly which version of a document contributed to a given response. Including a relevance score attribute is particularly useful during debugging, as it lets you verify that the model is preferring high-relevance sources as instructed.

A practical consideration: XML is verbose. Each <source> tag pair consumes tokens, and in a tight context budget, those tokens come at the cost of document content. For very large document sets or very small context windows, you may prefer a lighter delimiter scheme such as triple dashes or numbered headers. The right choice depends on your token budget and how much structural clarity you need.

The instructions for citation behavior must be precise. Vague instructions lead to inconsistent behavior, where the model sometimes cites, sometimes does not, and occasionally invents citations to documents that do not exist in the prompt. Consider this progression from vague to specific:

Specificity matters because models are sensitive to instruction following patterns established during fine-tuning. Ambiguous instructions lead to inconsistent citation behavior, either omitting necessary attributions or hallucinating citations to non-existent sources. The specific version eliminates ambiguity about syntax ([doc_001] not doc_001 or (1)), scope (every factual claim, not just major ones), multi-source handling (cite all supporting documents), and fallback behavior (what to say when context is insufficient).

There is a tension between instruction specificity and prompt length. Every additional sentence in your citation instructions consumes tokens that could otherwise hold retrieved content. For applications where every token matters, consider encoding citation instructions as few-shot examples rather than prose descriptions. A single example showing a properly cited response often communicates the pattern more efficiently than three paragraphs explaining it.

One technique that significantly improves citation accuracy is instructing the model to verify its citations before finalizing its response. The instruction might look like:

This self-verification step adds output tokens, but it activates a form of retrieval-augmented generation within the generation itself. The model loops back over its claimed citations, checking them against its representation of the source documents, and corrects errors before they reach the user. The technique is most effective with larger models that have sufficient capacity to maintain accurate representations of all source documents simultaneously.

Even with efficient retrieval and reranking, you will frequently encounter situations where the combined retrieved documents exceed your model's context window. Context windows have expanded dramatically in recent years, but so has the volume of retrievable content. A system retrieving from a million-document corpus may surface twenty highly relevant passages, each thousands of tokens long, for a total that vastly exceeds even a 128K-token context window. When truncation is necessary, how you truncate matters immensely: a poorly truncated document that cuts off its key sentence is worse than no document at all, because it occupies space while providing no useful information.

Think of truncation as budget allocation under uncertainty. You have a fixed token budget for all retrieved content, and you must distribute it across documents in a way that maximizes the expected relevance of what you keep. Different truncation strategies reflect different assumptions about where useful information lives within a document. The inverted pyramid assumption (information density decreases from beginning to end) favors keeping the head. The conclusions-first assumption (answers live in summaries and conclusions) favors keeping the tail. The query-centric assumption (relevant information can appear anywhere, but clusters around query-similar sentences) favors sliding window extraction.

The mathematical challenge of truncation involves optimizing an information preservation function subject to a constraint on token count. Given a document $d$ consisting of tokens $t_1, t_2, \ldots, t_n$, and a maximum token budget $k$, we seek to extract a subsequence $d'$ that maximizes the relevance function $R(d', q)$ with respect to query $q$, subject to the constraint $|d'| \leq k$.

Formally, this is a constrained optimization problem:

where:

• $d'$: the extracted subsequence of document tokens, a subset of $d$
• $R(d', q)$: the semantic relevance of the extracted passage to the query $q$, estimated by embedding similarity or another scoring function
• $k$: the token budget, a hard constraint determined by the context window size minus the space reserved for instructions and the user query

Different truncation strategies represent different approximations of this optimization problem based on assumptions about document structure and information distribution. None of them solve the problem exactly because computing $R(d', q)$ for all possible subsequences is combinatorially intractable. Instead, each strategy uses a structural heuristic to identify a tractable approximate solution.

The simplest approach keeps the beginning of each document and truncates the end:

This strategy operates on the assumption that information follows an inverted pyramid structure, common in journalistic and academic writing, where the most salient content appears in the introduction. Mathematically, this assumes that the relevance function $R$ is monotonically decreasing with position index $i$:

This preserves introductions and thesis statements but loses conclusions and summaries. It works well for academic papers and news articles where the inverted pyramid structure places key information early. It fails badly for technical documentation structured as problem-solution pairs where the solution appears at the end, or for question-answer formatted content where the answer only appears after a lengthy preamble.

Head-only truncation is also computationally simple: it requires no embeddings, no sentence segmentation, and no query awareness. This makes it the default choice for many production systems despite its obvious limitations, because it is the only strategy that adds zero latency. When you are serving thousands of requests per second, even a few milliseconds of additional latency from smarter truncation can matter.

Conversely, keeping only the end of documents preserves conclusions and final summaries:

This approach inverts the monotonicity assumption, positing that $R(t_i, q) \leq R(t_{i+1}, q)$. This suits documents where the final paragraphs contain the synthesis or answer, such as FAQ entries, legal briefs where the conclusion of law appears at the end, or research papers where the results section follows the methodology. The key risk with tail truncation is that it often discards the context necessary to interpret the kept content. A conclusion that says "therefore, the treatment was effective" without the preceding description of what treatment was tested provides little actionable information.

For long documents where relevant information might appear anywhere, a sliding window approach extracts segments around query-relevant sentences:

When individual documents are too long for simple truncation, hierarchical approaches create distillations that preserve more of the semantic content. The idea is to use a fast, cheap model to compress each document before including it in the main prompt, trading some detail for coverage.

The process works in three stages. First, you summarize each long document independently using either an extractive method (selecting the most relevant sentences) or an abstractive method (generating a shorter paraphrase). Second, you use these summaries as the context instead of the full text, allowing you to include information from many more documents within the same token budget. Third, for the top-scoring documents, you may optionally include the full text alongside the summary, giving the model access to fine-grained detail where it matters most.

This approach trades granularity for coverage: you can represent twenty documents instead of five, but each representation is lossy. The most significant risk is that the summarization step introduces its own errors. A summary model that misses a key qualifier, such as "the drug was effective only in patients over 65," can lead the main model to generate a response that omits critical nuance. Summarization before retrieval, at indexing time, is generally more reliable than summarization at inference time because it can be validated and corrected before it affects users.

The system prompt or task instruction in a RAG pipeline carries additional responsibility beyond standard task definition. It must establish the relationship between the retrieved context and the generation task, define what counts as an authoritative source, specify citation behavior, and handle edge cases gracefully. A well-designed instruction turns a general-purpose language model into a reliable, attributable research assistant. A poorly designed instruction produces confident-sounding responses that ignore the retrieved context or fabricate plausible-looking citations.

Think of the instruction as a constitution for your RAG system. It specifies the rules that govern how the model treats the retrieved documents: do they take precedence over the model's parametric knowledge? What should the model do when documents contradict each other? What happens when no relevant documents were retrieved? Getting these rules right is what separates a RAG system that can be trusted in production from one that works well on a demo but fails on edge cases.

The instruction also mediates a fundamental tension in RAG design. You want the model to use the retrieved context faithfully, but you do not want it to be so literal that it fails to synthesize, draw inferences, or connect information across documents. A model that only parrots back text from the retrieved passages is not adding value; a model that synthesizes those passages into a coherent, well-reasoned answer is what makes RAG worth the engineering effort.

Effective RAG instructions establish a clear contract with the model. This contract has four components, each addressing a different failure mode.

The first component is authority: explicitly stating that the retrieved context takes precedence over parametric knowledge prevents the model from substituting its training data when the context says something unexpected or unfamiliar. The second component is scope: defining what the model should do if the context is insufficient prevents hallucination when the retrieval fails to find relevant documents. The third component is constraints: specifying the required format and tone, along with citation rules, ensures the output is usable for downstream processes. The fourth component is fallbacks: providing behavior for contradictory or irrelevant context prevents the model from picking one source arbitrarily or blending conflicting claims without flagging the conflict.

Consider this instruction template:

The numbered list format is intentional. Research on instruction following shows that numbered lists receive more reliable compliance than prose paragraphs, likely because the model has seen many examples of numbered instructions in its training data and has learned to treat each number as a discrete constraint to satisfy. Prose instructions blend together in generation, and the model may satisfy some while forgetting others.

Production RAG systems encounter edge cases that naive prompt designs handle poorly. Addressing these explicitly in the instruction template is far easier than debugging them after deployment.

The empty retrieval edge case occurs when no documents are retrieved above a relevance threshold, typically because the query is out-of-domain or the index does not contain relevant material. Without an explicit instruction, the model will often generate a response from parametric knowledge, making it look like the system found something when it found nothing. The instruction should tell the model to explicitly decline or request clarification: "If no relevant sources were retrieved, respond: 'I could not find information about this topic in the available sources.'"

The contradictory sources edge case occurs when retrieved documents disagree with each other. This is common in rapidly evolving fields, in domains with contested interpretations (law, medicine, policy), and when documents from different time periods coexist in the index. Without an explicit instruction, the model may arbitrarily favor one source, blend the contradictory claims into a false synthesis, or silently ignore one side of the disagreement. The instruction should direct the model to state the contradiction: "If sources disagree, explicitly note the disagreement and cite all conflicting sources."

The outdated information edge case arises when the retrieved documents contain information that was once correct but is no longer current. This is particularly dangerous in fast-moving domains where a two-year-old document about a regulation or a technology product may be significantly misleading. Including timestamp metadata in your source formatting and instructing the model to prefer more recent sources when temporal relevance matters provides a first line of defense, though it does not fully solve the problem.

Let's walk through a concrete example to see how all these principles come together in a realistic setting. Imagine you are building a RAG system for legal research where users ask about case law precedents.

The query: "What constitutes fair use for educational purposes in the Second Circuit?"

The retrieved documents:

1. Campbell v. Acuff-Rose Music (Supreme Court, 1994), 800 tokens discussing transformative use
2. NXIVM Corp. v. Ross Institute (2nd Circuit, 2004), 600 tokens on commercial vs. educational use
3. Authors Guild v. HathiTrust (2nd Circuit, 2014), 1200 tokens on digital library scanning
4. Cambridge University Press v. Patton (11th Circuit, 2016), 900 tokens on e-reserves (a different circuit but relevant)

Total retrieved content: 3500 tokens, which fits comfortably in most context windows. But the documents have heterogeneous authority: two are binding Second Circuit precedent (documents 2 and 3), one is persuasive Supreme Court authority (document 1), and one is persuasive authority from a different circuit (document 4). A naive prompt treats them all equally; a well-designed prompt respects this hierarchy.

Strategy A: Naive Concatenation

This strategy has multiple problems. There are no citation markers, so the model cannot attribute claims to specific cases. There is no indication of authority hierarchy, so the model may treat a persuasive 11th Circuit case as equally controlling to Second Circuit precedent. The most relevant documents (2nd Circuit cases) are buried in the middle, right in the lost-in-the-middle zone. And there is no instruction about what to do if the cases conflict.

Strategy B: Structured with Prioritization

This structure addresses each problem in Strategy A. The most relevant Second Circuit cases are placed at the beginning and end of the sources section (HathiTrust first, NXIVM last), exploiting the primacy and recency zones. Documents are labeled with meaningful identifiers that encode authority level. The instructions explicitly specify the authority hierarchy and citation format. And the fallback behavior for conflicting cases (prefer most recent Second Circuit) removes ambiguity.

The structured version is longer by roughly 100 tokens of overhead (labels, section headers, instructions), but the investment pays for itself in response quality. The model knows exactly which cases are controlling, how to cite them, and what to do when they point in different directions. The result is a response that reads like it was written by a lawyer, not a general-purpose language model.

Numerical breakdown: The total token budget is 4100 (3500 retrieved content + 600 overhead). The sandwich ordering places the 1200-token HathiTrust case (highest relevance, most recent 2nd Circuit precedent) first, which means it lands in the high-attention primacy zone. The 600-token NXIVM case (second-highest relevance, controlling 2nd Circuit precedent) goes last, landing in the recency zone. The 300-token truncated versions of documents 1 and 4 go in the middle, where their lower authority makes the attention deficit less costly. This is the sandwich pattern applied with domain knowledge: the most authoritative and most relevant documents get the premium seats.

Let's implement a flexible RAG prompt construction system that incorporates the principles we have discussed throughout this chapter.

The output shows our structured XML formatting with relevance scores, strategic ordering (highest relevance documents at boundaries), and weighted truncation that allocates more space to higher-scoring documents. Notice how doc_001 (score 0.95) appears first in the sources section and doc_002 (score 0.88) appears last, implementing the sandwich pattern. The weighted truncation gives doc_001 roughly twice as many characters as doc_004. This reflects the 2:1 ratio of their scores.

The key parameters for the RAG prompt builder form a configuration space that determines the information-theoretic properties of the final context window. These parameters interact in ways that affect both the computational complexity and the semantic fidelity of the retrieval augmentation:

The parameters interact in non-obvious ways. A large max_context_tokens budget combined with "uniform" truncation may hurt performance if it means every document gets truncated to the same short length, losing the detail in the most relevant documents. A smaller budget with "weighted" truncation may produce better results because the highest-relevance documents receive enough space to present their content coherently. Similarly, "chronological" ordering is only useful when your retrieval scores are not well-calibrated for temporal relevance; if your retriever already down-weights old documents appropriately, chronological ordering just overrides that calibration.

RAG prompt engineering has fundamental constraints that practitioners must manage carefully. Understanding these limitations is essential for setting realistic expectations, designing appropriate fallback behaviors, and knowing when to invest in additional infrastructure versus accepting the inherent tradeoffs of the approach.

As context length grows, even with careful placement, the fidelity of processing degrades because attention bandwidth is limited. Each additional retrieved document competes for attention weights with every other document and with the query itself. When you include twenty retrieved passages, each one receives, on average, five percent of the attention energy that a single document would receive in isolation. This dilution means that subtle distinctions in source material get smoothed over or lost entirely.

The problem compounds with the nature of retrieved content. Unlike carefully curated training examples, retrieved documents often contain redundant information, boilerplate text, and irrelevant sections. A court case retrieved for its discussion of transformative use might also contain ten pages of procedural history and attorney billing disputes. Without aggressive deduplication and extractive preprocessing, you waste precious attention capacity on repetitive or irrelevant content. This is why the preparation of retrieved content, stripping boilerplate, deduplicating near-identical passages, and normalizing formatting, is often as important as the retrieval quality itself.

The practical implication is to use fewer, better documents rather than many mediocre ones. A RAG system that retrieves 5 high-quality passages will typically outperform one that retrieves 20 mixed-quality passages, because the attention dilution in the 20-document case degrades even the good passages. Set retrieval count limits that reflect your model's effective attention capacity, not just its nominal context window size.

Language models possess parametric knowledge acquired during pre-training, and this knowledge can override retrieved context, especially when the retrieval contradicts the model's training data. If your retrieved document states that the capital of Australia is Sydney (incorrect), but the model's training data strongly associates Australia with Canberra, the model may ignore the retrieved text and rely on its parametric knowledge, or worse, blend the two into a confused response.

This parametric override problem is particularly acute in specialized or technical domains where the model has strong prior beliefs from training. A medical RAG system retrieving a document about a newly approved drug dosage may find the model ignoring the correct retrieved dosage because its training data contained a different value. The model's confidence in its parametric knowledge is not calibrated to the date at which its knowledge becomes stale; it simply has a strong association and generates accordingly.

Mitigating this requires explicit instruction design that subordinates parametric knowledge to retrieved context, but this remains imperfect. The model cannot easily "un-know" what it learned during pre-training, and strong factual associations create gravitational wells that pull generated text away from retrieved sources. Techniques like grounding chains of thought ("First, identify the relevant claim in the sources. Then, quote it directly. Then, answer the question based on that quote") can help by forcing the model to explicitly anchor its reasoning in the retrieved text before generating, but they add latency and token cost. Fine-tuning the model on RAG-specific tasks offers the strongest mitigation by teaching it to prefer retrieved context over parametric knowledge in retrieval-augmented settings.

Even with explicit citation instructions, models frequently hallucinate citations, attributing claims to sources that do not support them or citing non-existent document IDs. This stems from the mismatch between the discrete, symbolic nature of citation markers and the continuous, statistical nature of language model generation. The model learns that citations appear after claims, but it does not maintain a perfect lookup table of which source contains which fact during generation. When generation pressure leads to a plausible claim, the model may attach a plausible-sounding citation that turns out to be wrong.

Citation hallucination is particularly insidious because it is hard to detect without post-hoc verification. A response with citations that look syntactically correct may pass a superficial review but contain multiple attribution errors. Users tend to trust cited responses more than uncited ones, which means citation hallucination can increase the harm of model errors by lending them an unwarranted veneer of authority.

Production systems often implement post-hoc verification: using the generated citations to retrieve the claimed sources and checking that they contain the attributed information. This is typically implemented as a secondary pass that runs after generation, extracting all citation markers, looking up the corresponding documents, and verifying that the cited text appears in or is entailed by the source. This adds latency and complexity but significantly improves reliability. For high-stakes applications (legal, medical, financial), this verification step should be considered mandatory rather than optional.

A subtler limitation is that RAG prompt templates are designed and evaluated on a distribution of queries, but individual queries can fall far outside that distribution. A template optimized for fact-lookup queries may perform poorly when users ask for synthesis, comparison, or multi-hop reasoning that requires connecting information across documents. A template designed for short passages may fail when documents are long and heterogeneous.

This means prompt engineering for RAG is an iterative process: evaluate the system, diagnose failures, and revise the prompt. You need systematic evaluation infrastructure, such as test suites with ground-truth citations, to detect when your prompt template is failing on a class of queries. Building that infrastructure early, before deployment, is far cheaper than discovering failures through user complaints. We will explore evaluation frameworks for RAG systems in the next chapter.

RAG prompt engineering bridges retrieval and generation, transforming retrieved documents into effective context through careful structural design. The quality of your prompt construction determines whether the retrieved information reaches the model in a form it can use reliably and cite accurately.

Strategic placement combats the lost-in-the-middle problem by positioning the most important documents at the beginning and end of the context window, following the U-shaped attention curve observed in large language models. This guards against a systematic failure mode that affects transformer-based models regardless of their nominal context window size. Designing around the lower attention paid to middle positions is safer than assuming a specific model handles long contexts perfectly.

Structured formatting using XML or clear delimiters helps models distinguish document content from its metadata and boundaries, reducing confusion between consecutive passages. The formatting choice also encodes provenance information: a source ID and date, plus a relevance score that the model can use when reasoning about which sources to trust and how to attribute claims. Consistent formatting within a prompt, using the same structure for every document, makes the model's job easier and reduces citation errors.

Explicit citation instructions establish clear contracts for attribution, though they require specificity and must be paired with verification mechanisms to prevent hallucinated citations. The more precisely you specify the citation format, the conditions under which to cite, and what to do when sources are insufficient, the more reliably the model will follow your instructions. Vague instructions produce vague compliance; specific instructions produce specific, auditable behavior.

Intelligent truncation allocates limited context capacity based on relevance, using strategies like weighted allocation or sliding window extraction to preserve the most query-relevant content. No truncation strategy is universally optimal; the right choice depends on your document corpus structure, your retrieval quality, and your latency budget. Building a system that supports multiple strategies and can be configured per use case is more valuable than optimizing a single strategy.

Edge case handling in system prompts prepares models for empty retrievals, contradictory sources, and authority hierarchies. These edge cases are not rare; in a production system serving diverse queries against a large corpus, they will occur daily. Designing for them upfront, rather than patching them after deployment, is the difference between a reliable system and one that fails in ways that are hard to predict or reproduce.

As we move toward more sophisticated agentic systems in later parts of this book, these RAG prompt engineering techniques form the foundation for systems that retrieve information and synthesize it with tool outputs and reasoning chains. The next chapter explores how we evaluate these complex RAG pipelines to ensure they meet accuracy and reliability standards in production environments.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about RAG prompt engineering.