Document Chunking: Optimizing RAG Retrieval Pipelines › Limitations and Practical Considerations

Despite the variety of strategies available, document chunking remains more art than science. No single strategy works optimally across all document types, query patterns, and embedding models. The fundamental challenge is that chunking decisions must be made at indexing time, before you know what questions you will ask. A chunk boundary that perfectly separates two topics for one query may split the exact passage needed for another. This temporal mismatch between when you chunk and when you retrieve is a structural property of the problem, not a bug that better algorithms can fully eliminate.

This points to a deeper limitation: the optimal chunking strategy is query-dependent, but chunking happens before any query is known. In an ideal world, you would chunk the same document differently for different queries: coarser chunks for broad analytical questions, finer chunks for specific factoid lookups, topic-aligned chunks for thematic questions. Practical systems cannot afford this luxury, so they must pick a strategy and chunk size that works acceptably across the distribution of expected queries. This is inherently a compromise, and it means that any production RAG system will have some queries it handles well and some it does not. Understanding this limitation helps you set realistic expectations and motivates investing in RAG evaluation infrastructure.

Semantic chunking, while conceptually appealing, introduces its own challenges. It requires running an embedding model over every sentence during indexing, which significantly increases preprocessing time and cost. For a corpus of one million documents, each with an average of 100 sentences, semantic chunking requires 100 million sentence embeddings before a single query can be answered. This is feasible with GPU infrastructure but impractical for small teams doing quick experiments. The quality of the splits depends heavily on the embedding model's ability to capture topic coherence, and models trained primarily on sentence similarity may not always detect document-level topic transitions accurately. Semantic chunking can also produce highly variable chunk sizes: a long section on a single topic might produce a single enormous chunk, while a passage that quickly surveys several topics might be split into tiny fragments. Managing these extremes with minimum and maximum size constraints is possible, but those constraints partially undermine the semantic purity of the approach.

Another practical limitation is the interaction between chunk size and the downstream LLM's context window. If you retrieve five chunks of 500 tokens each, you consume 2,500 tokens of the LLM's context for retrieved content alone. With larger chunk sizes or more retrieved results, you may exhaust the context window before including your query and system instructions. This creates a system-level constraint: chunk size, number of retrieved results, prompt template length, and LLM context window all interact. A naive approach that optimizes chunk size in isolation, without considering how many chunks you plan to retrieve, can produce a system where the retrieval quality is excellent but the final LLM response is degraded by an overloaded context window.

Finally, evaluation of chunking quality is inherently tied to end-to-end RAG performance. You cannot evaluate chunking in isolation because the same chunks might work well with one embedding model and poorly with another, or might retrieve perfectly but confuse a particular LLM. A chunk that is semantically clean but starts with a pronoun referencing an entity from the previous chunk ("It was first discovered...") may retrieve correctly but confuse the LLM about what "it" refers to. These cross-chunk coherence issues are invisible to chunking-level metrics and only surface in end-to-end evaluation. We will address this challenge systematically in the RAG Evaluation chapter later in this part.