Document Chunking: Optimizing RAG Retrieval Pipelines › Chunk Size Selection
Language AI HandbookAll chapters Document Chunking: Optimizing RAG Retrieval Pipelines › Chunk Size SelectionOne section · Sections in this chapter7 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Choosing the right chunk size is one of the most impactful decisions in a RAG pipeline. It involves a fundamental trade-off between precision and context, and the optimal balance depends on the nature of your data, the types of questions you ask, and the capabilities of your embedding model. Getting chunk size right can mean the difference between a system that reliably retrieves the right passage and one that returns vaguely related blocks of text.
Think of chunk size selection as choosing the magnification level on a microscope. A very high magnification gives you fine detail, but you lose the ability to see how structures relate to one another within the broader tissue. A low magnification shows you the big picture but reveals little about individual cells. The right magnification depends on what you are trying to see. Similarly, chunk size must be chosen based on the granularity of questions you expect users to ask and the granularity of information that documents contain.
Before examining the formal trade-off, it helps to understand the two failure modes at the extremes. When chunks are too small, retrieval becomes fragile: you can find the right words, but the surrounding context is missing, and the LLM cannot produce a useful answer. When chunks are too large, retrieval becomes imprecise: the embedding represents multiple topics, and the cosine similarity between query and chunk reflects a blend of relevance rather than a strong match. Understanding these failure modes helps you diagnose problems in your own system.
The Precision-Context Trade-off
The precision-context trade-off is the central tension in chunk size selection. At one extreme, you could make each chunk a single sentence, maximizing precision. At the other, you could embed entire documents, maximizing context. Neither extreme works well in practice, and understanding why illuminates the core challenge.
Small chunks (100-200 tokens) provide high retrieval precision. Each chunk covers a narrow topic, so when it matches a query, the match is likely relevant. The embedding vector represents a focused semantic concept, and the cosine similarity between query and chunk is a reliable indicator of topical alignment. However, small chunks may lack sufficient context for the LLM to generate a good answer. A chunk containing "The temperature was 42°C" is useless without knowing what system or location it refers to. The LLM receives a decontextualized fact and must either hallucinate the missing context or produce an unsatisfying response.
Large chunks (500-1,000 tokens) provide rich context. The LLM receives enough surrounding information to understand and synthesize an answer. A paragraph that introduces a concept, provides evidence, and draws a conclusion gives the LLM everything it needs to produce a coherent response. But large chunks produce less precise embeddings because they cover more topics, and they may include irrelevant information that confuses the model. When a 800-token chunk discusses three related but distinct subtopics, its embedding becomes a centroid in the semantic space that lies between all three topics. A query that is specifically about one of those subtopics may find a better cosine similarity match with a more focused chunk from a completely different document.
The optimal chunk size depends on several factors:
- Query type: Factoid questions ("What is the boiling point of water?") benefit from small chunks, while analytical questions ("Explain the causes of the 2008 financial crisis") need larger chunks with more context. If your application primarily serves one type of query, you can tune chunk size accordingly. If it serves a mix, you may need to compromise or use multiple chunk sizes.
- Document type: Technical documentation with short, self-contained sections works well with smaller chunks. Narrative text where ideas develop over paragraphs needs larger chunks. A software API reference, where each function's description is independent, naturally lends itself to small chunks. A legal brief, where arguments build across paragraphs, demands larger ones.
- Embedding model capacity: Different models have different optimal input lengths. Some models are trained on short passages and perform best with 1-2 sentences, while others handle full paragraphs effectively. Feeding a paragraph-optimized model a single sentence wastes its capacity, while feeding a sentence-optimized model a full paragraph may degrade its embedding quality.
Empirical Comparison
Let's compare how different chunk sizes affect the content of chunks from the same document. By viewing the same text through different chunk size lenses, we can build intuition for how this parameter shapes the granularity and coherence of the resulting pieces.
long_text = """
Machine learning is a subset of artificial intelligence that focuses on building systems
that learn from data. Unlike traditional programming where rules are explicitly coded,
machine learning algorithms identify patterns in data and make predictions based on
those patterns. The field has grown rapidly since the 2010s, driven by increases in
computational power and the availability of large datasets.
Supervised learning is the most common paradigm. In supervised learning, the algorithm
is trained on labeled examples, where each input is paired with its correct output.
Common supervised learning tasks include classification, where the goal is to assign
inputs to discrete categories, and regression, where the goal is to predict a continuous
value. Popular algorithms include linear regression, decision trees, and neural networks.
Unsupervised learning works with unlabeled data. The algorithm must discover structure
in the data without guidance. Clustering algorithms like K-means group similar data
points together. Dimensionality reduction techniques like PCA find compact representations
of high-dimensional data. Unsupervised learning is often used for exploratory data
analysis and feature extraction.
Reinforcement learning involves an agent that learns by interacting with an environment.
The agent takes actions and receives rewards or penalties based on the outcomes. Over time,
the agent learns a policy that maximizes cumulative reward. Reinforcement learning has
achieved notable successes in game playing, robotics, and most recently in aligning
large language models through RLHF, as discussed in earlier chapters of this book.
""".strip()size_results = {}
chunk_sizes = [100, 250, 500]
for size in chunk_sizes:
size_results[size] = chunk_by_sentences(long_text, max_chunk_size=size)=== Chunk size: 100 chars → 16 chunks === Chunk 0: [110 chars] Machine learning is a subset of artificial intelligence that focuses on building... Chunk 1: [164 chars] Unlike traditional programming where rules are explicitly coded, machine learnin... Chunk 2: [127 chars] The field has grown rapidly since the 2010s, driven by increases in computationa... Chunk 3: [48 chars] Supervised learning is the most common paradigm.... Chunk 4: [121 chars] In supervised learning, the algorithm is trained on labeled examples, where each... Chunk 5: [180 chars] Common supervised learning tasks include classification, where the goal is to as... Chunk 6: [82 chars] Popular algorithms include linear regression, decision trees, and neural network... Chunk 7: [48 chars] Unsupervised learning works with unlabeled data.... Chunk 8: [67 chars] The algorithm must discover structure in the data without guidance.... Chunk 9: [70 chars] Clustering algorithms like K-means group similar data points together.... Chunk 10: [99 chars] Dimensionality reduction techniques like PCA find compact representations of hig... Chunk 11: [89 chars] Unsupervised learning is often used for exploratory data analysis and feature ex... Chunk 12: [88 chars] Reinforcement learning involves an agent that learns by interacting with an envi... Chunk 13: [80 chars] The agent takes actions and receives rewards or penalties based on the outcomes.... Chunk 14: [70 chars] Over time, the agent learns a policy that maximizes cumulative reward.... Chunk 15: [193 chars] Reinforcement learning has achieved notable successes in game playing, robotics,... === Chunk size: 250 chars → 10 chunks === Chunk 0: [110 chars] Machine learning is a subset of artificial intelligence that focuses on building... Chunk 1: [164 chars] Unlike traditional programming where rules are explicitly coded, machine learnin... Chunk 2: [176 chars] The field has grown rapidly since the 2010s, driven by increases in computationa... Chunk 3: [121 chars] In supervised learning, the algorithm is trained on labeled examples, where each... Chunk 4: [180 chars] Common supervised learning tasks include classification, where the goal is to as... Chunk 5: [199 chars] Popular algorithms include linear regression, decision trees, and neural network... Chunk 6: [170 chars] Clustering algorithms like K-means group similar data points together. Dimension... Chunk 7: [178 chars] Unsupervised learning is often used for exploratory data analysis and feature ex... Chunk 8: [151 chars] The agent takes actions and receives rewards or penalties based on the outcomes.... Chunk 9: [193 chars] Reinforcement learning has achieved notable successes in game playing, robotics,... === Chunk size: 500 chars → 4 chunks === Chunk 0: [452 chars] Machine learning is a subset of artificial intelligence that focuses on building... Chunk 1: [434 chars] In supervised learning, the algorithm is trained on labeled examples, where each... Chunk 2: [498 chars] The algorithm must discover structure in the data without guidance. Clustering a... Chunk 3: [264 chars] Over time, the agent learns a policy that maximizes cumulative reward. Reinforce...
Smaller chunk sizes produce more chunks, each covering a narrower topic. With a 100-character limit, individual sentences or pairs of short sentences become the unit of retrieval, giving the system laser-like precision but very little surrounding context. Larger chunk sizes produce fewer chunks that span multiple topics: a 500-character chunk might combine the definition of supervised learning with examples of specific algorithms, creating a richer but less focused unit.
The key insight here is that neither precision nor context is a fixed quantity: both are determined by the relationship between chunk size and document structure. A 250-character chunk of a dense academic paper might be too small to convey a complete argument, while a 250-character chunk of a FAQ document might contain an entire question-answer pair. The same number means different things in different documents.
There is no universally correct size; the right choice depends on your use case and should ideally be determined through evaluation. Most practitioners start with 256-512 tokens as a reasonable default, then adjust based on observed retrieval quality. We will cover systematic evaluation in the RAG Evaluation chapter.