RAG Architecture: Components, Timing & Design Patterns › Key Parameters
Michael BrenndoerferJanuary 21, 20262 min read
Language AI HandbookAll chapters Part 44 of 67Retrieval-Augmented Generation
RAG Architecture: Components, Timing & Design Patterns › Key ParametersOne section · Sections in this chapter8 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding the key parameters in a RAG pipeline allows you to tune the system for your specific requirements. These parameters interact with each other, and changing one often requires re-tuning others.
The most important parameters are:
- k: The number of documents to retrieve. Balances context coverage against noise. Typical values range from 3 to 10, with lower values for focused queries and higher values for broad research questions. Larger increases latency both from the retrieval step (more results to rank) and the generation step (longer context to process). Experiment with values from 1 to 20 and measure answer quality to find the sweet spot for your domain.
- score_threshold: The minimum similarity score required to consider a document relevant. This threshold depends heavily on the retrieval method and corpus, and should be tuned on representative queries. A threshold calibrated on one corpus may not transfer to another.
- max_tokens: The maximum number of tokens to generate in the response. Longer limits allow more complete answers but increase latency and cost. Consider the typical answer length for your domain when setting this parameter.
- chunk_size: The size of document chunks in tokens. Smaller chunks (128-256 tokens) are better for precise factual retrieval; larger chunks (512-1024 tokens) preserve more context and work better for questions requiring understanding of extended passages.
- chunk_overlap: The number of tokens that adjacent chunks share. Overlap of 10-20 percent reduces the chance of splitting relevant content across chunk boundaries at the cost of some index redundancy.