RAG Architecture: Components, Timing & Design Patterns › The Retriever Component

The retriever's job is deceptively simple: given a query, return the documents most likely to contain relevant information. However, this process is complex. The retriever must somehow measure the relevance between your question, expressed in natural language with all its ambiguity and variation, and documents that may discuss the same concepts using entirely different vocabulary. Someone asking about "automobile reliability" and a document discussing "car dependability" are clearly related, but a keyword matcher would miss the connection entirely. In practice, achieving effective retrieval involves three sub-systems working together: document encoding, indexing, and query-time retrieval. Each sub-system has its own design decisions, and getting them right requires understanding both the algorithmic options and the practical constraints of your deployment environment.

The retriever is often the weakest link in RAG systems. Language model generators have improved dramatically over the past few years, but retrieval remains a hard problem. The challenge is that relevance is inherently query-dependent: the same document might be highly relevant for one question and irrelevant for another. A document explaining how attention mechanisms work in transformers is invaluable for someone asking "how does self-attention work?" but useless for someone asking "what is the capital of France?" The retriever must make relevance judgments without knowing exactly what the generator needs, and it must make those judgments quickly enough to support interactive applications.

Think of the retriever as a sommelier at a restaurant. When you describe what you'd like to drink, the sommelier doesn't pour every bottle in the cellar, they draw on their knowledge of the wine list to select a few bottles they believe will satisfy you. Their recommendations depend on how well they understood your description, how well they know the cellar, and how good their model of your preferences is. A great sommelier finds exactly what you wanted. A mediocre one gives you something in the right category but misses the details. The retriever faces an analogous challenge: it must interpret your query and search a knowledge base to find the items most likely to satisfy the generator.

Before retrieval can happen, documents must be transformed into a searchable representation. Raw text, while meaningful to humans, cannot be efficiently compared or searched by algorithms. The encoding process bridges this gap, converting human-readable text into mathematical objects that computers can manipulate. This process typically involves two stages that work together to prepare documents for rapid, accurate retrieval.

First, documents are split into smaller chunks. A 50-page technical report cannot be fed to a language model in its entirety, so we divide it into passages of a few hundred tokens each. The chunking strategy significantly affects retrieval quality: too small and you lose context, too large and you dilute relevant information with noise. Consider a chunk that contains only half a sentence: the retriever might match it to a query based on keyword overlap, but the generator would receive incomplete, potentially misleading information. Conversely, a chunk spanning multiple pages might contain one relevant paragraph buried among irrelevant content, which makes it harder for the generator to identify what matters. A common heuristic is 256 to 512 tokens per chunk with 10 to 20 percent overlap between adjacent chunks, but the right size depends on your documents and queries. Technical documentation with dense, self-contained paragraphs might warrant smaller chunks, while narrative text with ideas that span multiple paragraphs might need larger ones. We'll explore chunking strategies in depth in a later chapter, examining techniques like overlapping windows, semantic boundary detection, and hierarchical chunking.

Second, each chunk is converted into a numerical representation suitable for similarity search. This transformation is the heart of the retrieval process, determining how the system measures whether one piece of text relates to another. Two broad approaches dominate the field, each with distinct characteristics that make them suited to different scenarios.

Sparse representations like TF-IDF and BM25 represent documents as high-dimensional vectors where each dimension corresponds to a vocabulary term. As we covered in Part II, BM25 scores documents based on term frequency, document length normalization, and inverse document frequency. The intuition behind sparse representations is straightforward: documents are characterized by the words they contain. A document about neural networks will have non-zero values for terms like "neuron," "layer," and "activation," while a document about cooking will have non-zero values for entirely different terms. These representations are sparse because most terms don't appear in any given document, leaving most dimensions at zero. A vocabulary might contain 100,000 terms, but any single document uses only a few hundred, resulting in vectors that are more than 99 percent zeros. The sparsity is not a weakness, it enables the highly efficient inverted index data structure that makes BM25 retrieval lightning fast. Sparse methods are interpretable (you can see exactly which terms caused a document to rank highly), require no training data, and can be deployed with minimal infrastructure. They remain competitive baselines and are often used in hybrid systems.

Dense representations use neural networks to encode chunks into continuous vectors, typically with 384 to 4096 dimensions. Unlike sparse vectors where dimensions correspond to specific words, dense embedding dimensions capture abstract semantic features that emerge during training. These features don't have human-interpretable names; instead, they represent learned patterns that help distinguish relevant from irrelevant content. Two documents discussing the same concept with different vocabulary will have similar dense embeddings even if their sparse representations share few non-zero dimensions. For example, a query about "car maintenance" would find documents discussing "vehicle repair" or "automobile servicing" because the neural network has learned that these phrases occupy similar regions of the embedding space. This semantic understanding comes at a cost: dense retrieval requires training a neural encoder (or fine-tuning a pre-trained one), storing and searching high-dimensional vectors, and GPU infrastructure for encoding at scale. But the quality gains are substantial, especially for queries that involve paraphrase or conceptual relationships rather than exact keyword matching.

The key insight is that sparse and dense representations capture complementary signals. Sparse methods are precise for exact terminology, dense methods are flexible for semantic content. This observation motivates hybrid retrieval, which we'll discuss in the architecture variations section.

The choice between sparse and dense retrieval involves trade-offs we'll examine in the next chapter. Sparse methods run quickly without training data, and their results are easy to interpret, making them excellent baselines. Dense methods capture semantic similarity that keyword matching misses but require substantial training data and computational resources. For now, recognize that both approaches solve the same basic problem: converting text into vectors that can be compared for similarity.

Encoded documents are stored in an index optimized for fast similarity search. Without an index, finding the most similar documents to a query would require computing similarity with every document in the collection. For a knowledge base with millions of documents, this brute-force approach would take seconds or minutes per query, far too slow for interactive applications. The index structure depends on the retrieval approach, with each type of representation requiring specialized data structures that exploit the mathematical properties of the representation.

For sparse retrieval, inverted indexes map each vocabulary term to the list of documents containing that term. The name "inverted" reflects that instead of mapping documents to their terms (the natural way we think about a document), we map terms to their documents. When a query arrives, the system looks up query terms, retrieves candidate documents containing those terms, and scores them using BM25 or similar formulas. This is the same technology powering web search engines, refined over decades to handle billions of documents with sub-second query times. The inverted index dramatically reduces computation because most queries contain only a handful of terms, and each term appears in only a fraction of documents. If your query contains three terms and each term appears in 0.1 percent of a 1 million document corpus, you need to score at most 3,000 documents rather than 1 million. That is a 99.7 percent reduction in computation from a simple data structure.

For dense retrieval, vector indexes organize embeddings to enable efficient nearest-neighbor search. Comparing a query embedding against millions of document embeddings naively requires millions of distance calculations, each involving hundreds or thousands of floating-point operations. Specialized data structures like Hierarchical Navigable Small World graphs (HNSW) and Inverted File indexes (IVF) reduce this to thousands or hundreds of comparisons through clever approximations. HNSW builds a multi-layer graph where each node connects to its nearest neighbors, letting search move quickly toward the most relevant region of the embedding space. Think of it as a transit network: at the top layer you have a few high-speed connections that let you travel large distances quickly, while at the bottom layer you have local connections for fine-grained navigation. IVF partitions the embedding space into clusters (using k-means) and searches only the clusters closest to the query vector, reducing the number of direct comparisons dramatically. These approximations trade a small amount of accuracy for dramatic speed improvements, and in practice the quality loss is negligible while the speedup can be 100x or more. We'll cover these algorithms in detail in upcoming chapters on vector similarity search.

The index improves performance while encoding a particular view of the data. An inverted index assumes that matching documents to queries is primarily about term overlap. A vector index assumes that semantic similarity is the primary relevance signal. Hybrid indexes can combine both, maintaining both an inverted index for exact match and a vector index for semantic match. When you choose an index, you're making a statement about what "relevant" means in your application.

When your query arrives, the retriever executes a specific sequence of operations. First, it encodes the query using the same method applied to documents. For sparse retrieval, this means computing BM25 term weights for the query terms. For dense retrieval, this means passing the query through the same neural encoder that embedded the documents. This symmetry is needed: queries and documents must inhabit the same representation space for similarity comparisons to be meaningful. If you encode documents with one model and queries with a different model, the resulting vectors occupy different spaces and comparisons will be meaningless.

Second, the retriever searches the index for the $k$ most similar documents. For sparse retrieval, this involves looking up query terms in the inverted index and combining the results using BM25 scoring. For dense retrieval, this involves traversing the vector index to find the nearest neighbors of the query embedding. In both cases, the index structure turns what would be an exhaustive search into a targeted lookup.

Third, the retriever returns those documents along with their similarity scores. These scores serve multiple purposes: they determine the ranking order, they can filter out low-quality matches, and they can inform the generator about the relative confidence of different sources. A document with a BM25 score of 8.2 and a document with a score of 0.3 should not be weighted equally when constructing the generator's prompt.

The parameter $k$ (typically 3 to 10 documents) balances coverage against noise. Too few documents risk missing relevant information; too many dilute the generator's context with marginally relevant passages. Research has consistently found that retrieving more documents helps up to a point, after which quality plateaus or even degrades as irrelevant content confuses the generator. The optimal value of $k$ depends on your use case: simple factual questions might need only one or two relevant passages, while complex analytical queries might benefit from synthesizing information across many sources.

BM25 retrieves documents sharing query terms like "transformer" and "architecture." The scores reflect term overlap, frequency, and document length normalization as defined by the BM25 formula we covered in Part II. Notice that the highest-scoring document explicitly mentions both "transformer" and "architecture," while lower-ranked documents may match only one of these key terms or match through related vocabulary.

One subtlety that trips up many developers is that queries and documents are not always encoded identically, even when using the same encoder model. In asymmetric search, queries tend to be short, conversational questions while documents are longer, more formal passages. Some dense retrieval models are specifically trained with different prompting strategies for queries versus documents to bridge this distributional gap.

For example, the E5 family of embedding models prepends "query:" to queries and "passage:" to documents before encoding. The model has been trained to understand these prefixes as signals about the type of text being encoded, letting it to represent the query in a way that aligns well with relevant passages rather than with similar-looking queries. Models like BGE use instruction-tuning approaches where you can provide task-specific instructions like "Represent this sentence for searching relevant passages:". These seemingly minor formatting differences can substantially affect retrieval quality on asymmetric tasks, which is exactly the setup typical RAG systems use. When you use a pre-trained retrieval model, always read its documentation carefully to understand how it expects queries and documents to be formatted.