RAG Architecture: Components, Timing & Design Patterns › Worked Example: Tracing a Query Through the Full Pipeline

Let's trace a specific query through every stage of a sequential RAG system to make the abstract components concrete. We'll use the query: "What is the key innovation that allows transformers to train faster than RNNs?"

Stage 1: Query Encoding (Sparse). The query is tokenized and lowercased: ["what", "is", "the", "key", "innovation", "that", "allows", "transformers", "to", "train", "faster", "than", "rnns"]. Stop words ("what", "is", "the", "that", "to", "than") are typically removed for BM25: ["key", "innovation", "allows", "transformers", "train", "faster", "rnns"]. BM25 term weights are computed for each remaining term based on their IDF values across the corpus.

Stage 2: Index Lookup. The retriever looks up each query term in the inverted index. "transformers" might appear in 1,247 documents, "train" in 3,821 documents, "faster" in 892 documents, and "innovation" in 654 documents. The inverted index returns candidate documents for each term, and BM25 scores are computed for each candidate based on the formula we studied in Part II. For our knowledge base, the highest-scoring document contains the sentence "The key innovation of transformers is replacing recurrence with self-attention. While RNNs process sequences step-by-step, transformers process all positions in parallel during training."

Stage 3: Top-k Selection. The retriever ranks all candidate documents by BM25 score and returns the top $k = 3$. The returned documents discuss transformers, self-attention, and the comparison with RNNs, all directly relevant to the query.

Stage 4: Context Formatting. The three retrieved documents are formatted into numbered context blocks with the query appended, following the template shown earlier. The full prompt is approximately 350 tokens.

Stage 5: Generation. The language model receives the 350-token prompt and generates a response attending to all tokens in the context. The model identifies the passage "replacing recurrence with self-attention" and "process all positions in parallel during training" as the key information. The generated response synthesizes this into a coherent answer: "The key innovation letting transformers to train faster than RNNs is the replacement of sequential recurrence with parallel self-attention. RNNs must process tokens one at a time, preventing parallelization across sequence positions. Transformers compute attention over all positions simultaneously, letting full GPU utilization during training."

Stage 6: Verification. In a production system, you would compare the generated claims against the retrieved documents to verify grounding. The claim about "parallel self-attention" is directly supported by Document 6 in the knowledge base. The claim about "sequential recurrence" preventing parallelization is a reasonable inference from the same passage.

This trace reveals why each component matters. If the retriever had returned documents about cooking instead of transformers, the generator's answer would have been wrong no matter how capable it was. If the generator had ignored the context and hallucinated, it might have said something plausible but incorrect. The final answer quality depends on both components working correctly in sequence.