RAG Prompt Engineering: Context Placement & Citation Strategies › Instruction Design for RAG
Language AI HandbookAll chapters RAG Prompt Engineering: Context Placement & Citation Strategies › Instruction Design for RAGOne section · Sections in this chapter5 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The system prompt or task instruction in a RAG pipeline carries additional responsibility beyond standard task definition. It must establish the relationship between the retrieved context and the generation task, define what counts as an authoritative source, specify citation behavior, and handle edge cases gracefully. A well-designed instruction turns a general-purpose language model into a reliable, attributable research assistant. A poorly designed instruction produces confident-sounding responses that ignore the retrieved context or fabricate plausible-looking citations.
Think of the instruction as a constitution for your RAG system. It specifies the rules that govern how the model treats the retrieved documents: do they take precedence over the model's parametric knowledge? What should the model do when documents contradict each other? What happens when no relevant documents were retrieved? Getting these rules right is what separates a RAG system that can be trusted in production from one that works well on a demo but fails on edge cases.
The instruction also mediates a fundamental tension in RAG design. You want the model to use the retrieved context faithfully, but you do not want it to be so literal that it fails to synthesize, draw inferences, or connect information across documents. A model that only parrots back text from the retrieved passages is not adding value; a model that synthesizes those passages into a coherent, well-reasoned answer is what makes RAG worth the engineering effort.
The Context-Task Contract
Effective RAG instructions establish a clear contract with the model. This contract has four components, each addressing a different failure mode.
The first component is authority: explicitly stating that the retrieved context takes precedence over parametric knowledge prevents the model from substituting its training data when the context says something unexpected or unfamiliar. The second component is scope: defining what the model should do if the context is insufficient prevents hallucination when the retrieval fails to find relevant documents. The third component is constraints: specifying the required format and tone, along with citation rules, ensures the output is usable for downstream processes. The fourth component is fallbacks: providing behavior for contradictory or irrelevant context prevents the model from picking one source arbitrarily or blending conflicting claims without flagging the conflict.
Consider this instruction template:
You are a helpful assistant with access to a knowledge base.
INSTRUCTIONS:
1. Base your answer PRIMARILY on the provided sources below
2. If the sources are insufficient, clearly state what information is missing
3. Cite sources for all factual claims using [source_id] format
4. If sources contradict each other, present both views and explain the disagreement
5. Do not use knowledge outside the provided sources unless specifically asked
SOURCES:
{retrieved_context}
USER QUERY: {query}
Provide a clear, accurate answer following the instructions above.
The numbered list format is intentional. Research on instruction following shows that numbered lists receive more reliable compliance than prose paragraphs, likely because the model has seen many examples of numbered instructions in its training data and has learned to treat each number as a discrete constraint to satisfy. Prose instructions blend together in generation, and the model may satisfy some while forgetting others.
Handling Edge Cases in Instructions
Production RAG systems encounter edge cases that naive prompt designs handle poorly. Addressing these explicitly in the instruction template is far easier than debugging them after deployment.
The empty retrieval edge case occurs when no documents are retrieved above a relevance threshold, typically because the query is out-of-domain or the index does not contain relevant material. Without an explicit instruction, the model will often generate a response from parametric knowledge, making it look like the system found something when it found nothing. The instruction should tell the model to explicitly decline or request clarification: "If no relevant sources were retrieved, respond: 'I could not find information about this topic in the available sources.'"
The contradictory sources edge case occurs when retrieved documents disagree with each other. This is common in rapidly evolving fields, in domains with contested interpretations (law, medicine, policy), and when documents from different time periods coexist in the index. Without an explicit instruction, the model may arbitrarily favor one source, blend the contradictory claims into a false synthesis, or silently ignore one side of the disagreement. The instruction should direct the model to state the contradiction: "If sources disagree, explicitly note the disagreement and cite all conflicting sources."
The outdated information edge case arises when the retrieved documents contain information that was once correct but is no longer current. This is particularly dangerous in fast-moving domains where a two-year-old document about a regulation or a technology product may be significantly misleading. Including timestamp metadata in your source formatting and instructing the model to prefer more recent sources when temporal relevance matters provides a first line of defense, though it does not fully solve the problem.