Document Chunking: Optimizing RAG Retrieval Pipelines

In the previous chapters, we built up the core RAG pipeline: retrieving relevant documents and using them to ground an LLM's responses. But we glossed over a critical question: what exactly constitutes a "document" in retrieval? A single PDF might contain hundreds of pages. A webpage might span thousands of words. An internal knowledge base might hold gigabytes of technical documentation, legal contracts, and customer support transcripts. If you embed an entire document as one vector, the resulting embedding must compress an enormous amount of information into a single point in vector space. Important details get diluted, and retrieval precision suffers drastically.

Document chunking solves this problem by splitting large documents into smaller, self-contained pieces before embedding and indexing them. Think of it as deciding how to cut a long book into individual pages before filing them in a library catalog. The finer and more sensibly you cut, the easier it is for someone to find exactly the passage they need. The choice of chunking strategy, chunk size, and overlap between chunks has a surprisingly large impact on RAG quality. A poorly chunked document can bury relevant information inside irrelevant context, while a well-chunked document makes the needed passage easy to retrieve.

The importance of this decision is easy to underestimate. Many practitioners treat chunking as a preprocessing detail and move on quickly to more glamorous components like embedding models or vector databases. But chunking is foundational. It defines the fundamental unit of retrieval: what size and shape of knowledge can the system find and return? Get this wrong and no amount of fine-tuning the downstream components will fully compensate.

Consider a realistic scenario. You are building a RAG system over a collection of scientific papers. A user asks: "What was the sample size in the Jensen et al. study on working memory?" If you have chunked each paper into a single vector, the answer to that highly specific question is buried inside an embedding that also represents the abstract, the literature review, the discussion section, and the conclusions. The probability of retrieving the exact paper is reasonable, but the probability of retrieving the exact sentence containing the sample size is low. If instead you chunk at the paragraph level, the specific methodology paragraph containing the sample size becomes a distinct, findable unit in your index.

This chapter examines chunking strategies, from simple fixed-size splits to semantically aware approaches that respect the natural structure of text. We will implement each strategy, examine how chunk size affects retrieval, build intuition for when to use each approach, and work through a concrete numerical example that illustrates the mechanics of each method. Along the way, we will cover chunk overlap, metadata enrichment, and the special challenges posed by different document types.

To understand why chunking is so important, consider what happens during the retrieval stage of RAG. As we discussed in the Dense Retrieval chapter, we encode both queries and documents into dense vectors and retrieve documents whose embeddings are closest to the query embedding. The quality of this retrieval depends directly on how well each embedding captures the meaning of its text. If the embedding is a faithful representation of a focused, coherent passage, it will match queries about that passage with high confidence. If the embedding is a blurred summary of many disparate topics, it becomes a mediocre match for all of them and a strong match for none.

Think of an embedding as a GPS coordinate in semantic space. A chunk about "ocean temperature trends in the Arctic" gets a precise coordinate in the region of oceanography and climate science. A chunk that covers ocean temperatures, carbon policy, economic impacts, and indigenous communities gets a coordinate that averages all of those regions, landing somewhere in the fuzzy middle ground between them. When a query about Arctic ocean temperatures approaches that coordinate, it is farther from the document-level average embedding than from a focused chunk embedding. Precision degrades not because the information is absent, but because it is diluted.

Embedding models have two fundamental constraints that make chunking necessary. Understanding both helps you reason about what will go wrong if you ignore them.

The first constraint is the context window limit. Most embedding models have a maximum input length, typically 512 tokens for models like those in the BERT family and up to 8,192 tokens for newer models. Text beyond this limit is simply truncated and lost. This means that if you feed a 10,000-token document into a 512-token embedding model, nearly 95% of the document is silently discarded. The resulting embedding reflects only the opening passage, leaving the vast majority of the document completely unrepresented in your index. An entire chapter on quantum computing, starting after the 512th token of a textbook, would simply not exist in your retrieval system.

The second constraint is information density. Even within the context window, longer texts produce embeddings that represent a blend of all topics covered. A 5,000-word article about climate change might discuss ocean temperatures, carbon emissions, policy proposals, and economic impacts. A single embedding for this article would be a vague average of all these topics, matching none of them precisely. Think of this like trying to describe an entire meal with a single adjective: "savory" might loosely apply to the soup, the steak, and the roasted vegetables, but it captures the distinctive character of none of them. The embedding loses the distinctiveness that makes retrieval useful.

Chunking addresses both constraints simultaneously. By splitting documents into smaller pieces, each chunk fits within the model's context window and covers a focused topic. When you ask about ocean temperature trends, the chunk specifically discussing that topic will produce a much stronger similarity match than the full article's embedding would.

Retrieval quality depends on having the right information represented in a way that makes it discoverable. Chunking creates this representation. Each chunk becomes a distinct, findable unit in vector space, carrying a focused semantic signal that can align with a matching query. The rest of this chapter explains how to create those units well.

The simplest chunking strategy splits text into pieces of a fixed number of characters or tokens, regardless of where sentences or paragraphs begin or end. Despite its simplicity, fixed-size chunking is common in production RAG systems because it runs quickly and produces predictable boundaries that are easy to reason about. When you need to process millions of documents quickly and you want deterministic, reproducible behavior, fixed-size chunking is often the pragmatic choice.

Think of fixed-size chunking as using a cookie cutter on dough. The cutter produces uniform shapes, and you can process them quickly on an assembly line. The downside is that sometimes you cut through the chocolate chip or split the raisin, producing shapes that lack the distinctive feature you were hoping to capture. The uniformity is an asset for engineering; it is a liability for meaning.

Its uniformity also simplifies downstream engineering: every chunk consumes roughly the same amount of storage, embedding compute, and retrieval bandwidth. You can predict exactly how many chunks a document will produce, how much index storage you need, and how long embedding will take. For large-scale systems where these properties matter, this predictability is valuable. Many production systems use fixed-size chunking not because it is optimal, but because it is reliable enough and far easier to operate at scale than semantically aware alternatives.

Before diving into implementations, it is worth developing intuition for what "fixed-size" means in practice. Two natural choices are character counts and token counts, and they behave quite differently.

The most basic approach splits text every $n$ characters. This produces chunks of uniform length but can cut words and sentences in the middle.

The mechanism is straightforward: we define a window of size $n$ and slide it along the text, recording each window as a chunk. When overlap is specified, adjacent chunks share some content. Formally, if we denote the full text as a sequence of characters $T = t_1, t_2, \ldots, t_L$, and we set chunk size $n$ and overlap $o$, then chunk $k$ contains characters at positions:

where:

• $k$: the zero-indexed chunk number
• $n$: the chunk size in characters
• $o$: the overlap in characters (between 0 and $n$)
• $T[a:b]$: the substring of $T$ from position $a$ to $b$

The stride between consecutive chunks is $n - o$ characters. With zero overlap, $o = 0$, and chunks are perfectly non-overlapping; each position in the text appears in exactly one chunk. With positive overlap, positions in the overlap zone appear in two consecutive chunks.

Notice how chunks cut through words and sentences without regard for meaning. Chunk 1 might start in the middle of a word, making it difficult for an embedding model to capture the intended meaning. A phrase like "approximately 10 percent of all species" could be severed into "approxi" at the end of one chunk and "mately 10 percent of all species" at the beginning of the next, rendering both fragments less semantically coherent than the original sentence.

This is the fundamental weakness of character-based splitting: it optimizes for uniform size at the expense of coherence. The embedding model must then try to extract meaning from text that may begin or end mid-thought, producing embeddings that are noisier and less representative of any single idea. For use cases where speed and simplicity matter more than embedding quality, character-based splitting is a reasonable starting point. For use cases where retrieval precision is critical, it almost always warrants improvement.

A better variant splits by token count rather than character count. Since embedding models operate on tokens, not raw characters, this ensures each chunk uses the model's capacity efficiently. A character-based chunk of 200 characters might translate to anywhere from 30 to 60 tokens depending on word length and vocabulary, creating unpredictable utilization of the embedding model's context window. Token-based splitting eliminates this variability by directly controlling the unit that matters.

The key insight here is alignment: you want to control the input size in the same units the model uses internally. Controlling characters while the model measures tokens is like setting a cooking timer in Fahrenheit when the recipe is in Celsius. You can make it work, but you are adding unnecessary conversion error. Token-based splitting removes that friction.

We can use the tiktoken library, which implements the tokenizers used by OpenAI models, to perform this conversion accurately. The cl100k_base encoding is the tokenizer used by GPT-4 and GPT-3.5-turbo, and it is widely used as a proxy tokenizer even for non-OpenAI models because it has well-understood behavior on English text.

Token-based splitting guarantees each chunk uses a predictable number of tokens, but it still does not respect sentence boundaries. A sentence split across two chunks loses coherence in both. The first chunk ends with an incomplete thought, and the second begins without the context established earlier in the sentence. For many retrieval scenarios, this partial-sentence problem degrades embedding quality enough to motivate the sentence-aware approaches we explore next.

The key parameters for fixed-size chunking are:

• chunk_size: The target size of each chunk (in characters or tokens). Smaller chunks are more precise, while larger chunks provide more context. This parameter directly controls the trade-off between retrieval granularity and the amount of information each chunk carries.
• overlap: The number of units (characters or tokens) repeated between adjacent chunks to prevent information loss at boundaries. Overlap acts as a safety net. This keeps content near a cut point is fully represented in at least one chunk. We explore this concept in depth in the Chunk Overlap section below.

A more linguistically motivated approach uses sentence boundaries as the atomic unit. Rather than slicing text at arbitrary positions, this strategy first identifies where sentences end and then groups consecutive sentences together until a size limit is reached. The result is chunks that always begin and end at natural linguistic boundaries.

As we covered in the Sentence Segmentation chapter, identifying sentence boundaries is itself a non-trivial task that must handle abbreviations, decimal numbers, and other edge cases. A naive approach of splitting on periods fails for "Dr. Smith increased the dosage to 2.5 mg." Here, we use spaCy's sentence segmentation for reliable boundary detection. SpaCy's model combines statistical patterns with rule-based heuristics that handle most edge cases correctly.

The idea is simple but powerful: split the text into individual sentences first, then group consecutive sentences together into chunks until adding the next sentence would exceed the desired size limit. Think of sentences as Lego bricks. Each brick is a complete, self-contained unit. You can stack them together to form larger structures, but you never break a brick in the middle. This guarantees that no chunk will ever contain a partial sentence, and every chunk begins at the start of a sentence and ends at the conclusion of one.

This sentence-preservation property matters because embedding models are trained on complete, well-formed text. Most modern sentence embedding models fine-tune their weights on datasets of complete sentences or paragraphs. When you feed them truncated or fragmented text, the resulting embeddings are noisier and less semantically reliable. Preserving sentence boundaries aligns your chunks with the distribution of text the embedding model was trained on, producing higher-quality representations.

Each chunk now contains complete sentences. The text within each chunk is coherent, making it much easier for an embedding model to capture its meaning. When the model processes a sentence-aligned chunk, it encounters grammatically well-formed text with clear subjects, verbs, and objects, exactly the kind of input it was trained on. This alignment between the structure of training data and the structure of your chunks tends to produce higher-quality embeddings.

The chunk sizes are no longer perfectly uniform, since they depend on sentence lengths, but this is a worthwhile trade-off. A chunk that is 180 characters of coherent prose will almost always produce a better embedding than a chunk that is exactly 200 characters but begins with the fragment "tion efforts are critical to preserving." The embedding model extracts meaning from the semantics of the text, not from its length. Feeding it coherent input produces a coherent, useful embedding.

One practical consideration with sentence-based chunking is that it can produce chunks of highly variable size when sentence lengths vary dramatically. A document alternating between very short sentences ("DNA mutates.") and very long ones could produce chunks that range from 10 characters to 400 characters for the same max_chunk_size setting. This variability rarely causes problems for retrieval quality, but it is worth monitoring if you care about predictable indexing time or storage usage.

The key parameters for sentence-based chunking are:

• max_chunk_size: The maximum length (in characters or tokens) allowed for a chunk before a split is forced. Sentences are accumulated into the current chunk as long as this limit is not exceeded, so actual chunk sizes will vary depending on where the sentence boundaries fall relative to this ceiling.
• overlap_sentences: The number of full sentences to repeat at the beginning of the next chunk to preserve context. Unlike character or token overlap, sentence overlap guarantees that the shared content is always a complete, meaningful unit. This is particularly valuable when consecutive sentences contain co-references ("This process..." or "These results...") that would be difficult to interpret without the preceding sentence.

Real documents have hierarchical structure: sections, subsections, paragraphs, sentences, and words. Recursive chunking exploits this structure by attempting to split at the most meaningful boundary possible. The strategy embodies a simple but effective principle: always prefer splitting at the highest-level boundary that still produces chunks within the size limit.

Think of recursive chunking as the way you might physically organize a large collection of documents. You first try to separate them by major category (sections). If a category still has too many documents, you separate by subcategory (paragraphs). If a subcategory is still too large, you split by individual pages (sentences). Only as a last resort do you cut mid-page (characters). At each level, you prefer the coarser division, resorting to finer divisions only when necessary.

The algorithm tries a sequence of separators in order of decreasing granularity. If splitting by double newlines (paragraph boundaries) produces chunks that are small enough, it stops there. If not, it falls back to single newlines, then sentences, then words, then characters. This layered approach means the algorithm naturally adapts to different parts of a document. A section with short paragraphs will be split along paragraph boundaries, preserving each paragraph as a self-contained unit. A section containing a single long paragraph will be split at sentence boundaries within that paragraph.

This approach is popularized by LangChain's RecursiveCharacterTextSplitter and is one of the most widely used strategies in practice. Its popularity stems from the fact that it produces reasonable results across a wide variety of document types without requiring any domain-specific configuration beyond the choice of separators. For many teams building their first RAG system, recursive chunking is the right default choice: it respects document structure when that structure exists, and degrades gracefully to simpler splits when it does not.

The recursive approach respects paragraph boundaries when possible, falling back to finer-grained splits only when a paragraph exceeds the size limit. This preserves the document's logical structure in the chunks. The "Climate Change Overview" heading and its accompanying paragraph naturally form one chunk, while the "Impact on Ecosystems" section forms another.

The algorithm arrives at these natural divisions not because it understands the content, but because the paragraph boundaries encoded by double newlines happen to align with topical boundaries. This is a pattern that holds reliably across well-structured documents, whether those documents are Wikipedia articles, academic papers, or technical documentation. The recursive chunker exploits this alignment silently and automatically, which is why it works so well in practice without domain-specific tuning.

Recursive chunking differs from simple separator-based splitting. A simple paragraph splitter would split every double newline, even when two short consecutive paragraphs together fit within the chunk size limit. Recursive chunking instead merges adjacent short paragraphs into a single chunk, only splitting when the merged size would exceed the limit. This behavior is generally preferable because it reduces the total number of chunks and ensures each chunk carries enough content to be useful.

The key parameters for recursive chunking are:

• chunk_size: The hard limit on chunk size. The algorithm attempts to keep chunks under this limit while using the largest possible separators. This means the algorithm always prefers a paragraph-level split over a sentence-level split, as long as both produce chunks that fit within this budget.
• overlap: The number of characters to overlap between chunks. In recursive chunking, overlap is applied after the initial splitting pass, duplicating content from the end of one chunk into the beginning of the next.
• separators: An ordered list of strings used to split the text (e.g., ["\n\n", "\n", " ", ""]). The algorithm tries them in sequence to find the best split point. The ordering encodes your preference for which types of boundaries should be preserved. By placing paragraph separators first, you ensure the algorithm only resorts to sentence or word breaks when paragraph-level splits are insufficient.

When you split a document into non-overlapping chunks, information that spans a chunk boundary gets split across two chunks. A key fact might have its context in one chunk and its conclusion in the next. Neither chunk alone captures the complete idea, and retrieval may miss it entirely.

Consider a passage where one sentence introduces a concept and the very next sentence provides the critical detail you are searching for. If the boundary falls between these two sentences, the first chunk contains a setup without a payoff, and the second chunk contains an answer without its question. An embedding of either chunk alone may fail to match your query. Think of a jigsaw puzzle where the most important piece of the picture straddles two boxes: you cannot see what it depicts by looking at either box alone.

Chunk overlap solves this by including some text from the end of each chunk at the beginning of the next. If your chunk size is 200 tokens and your overlap is 50 tokens, then the last 50 tokens of chunk $k$ appear again as the first 50 tokens of chunk $k+1$. This means any passage of 50 or fewer tokens near the boundary is guaranteed to appear in full within at least one chunk.

More formally, for a document with $N$ total tokens, a chunk size of $n$, and an overlap of $o$, chunk $k$ covers token positions:

where:

• $k$: the zero-indexed chunk number
• $n$: the number of tokens per chunk
• $o$: the number of overlapping tokens between adjacent chunks
• $\text{tokens}[a:b]$: the token subsequence from position $a$ to $b$

The total number of chunks produced is approximately:

where:

• $N$: total number of tokens in the document
• $K$: total number of chunks
• $\lceil \cdot \rceil$: the ceiling function (rounding up to the nearest integer)

Notice that increasing $o$ increases $K$. With zero overlap, $K = \lceil N/n \rceil$. With overlap equal to half the chunk size ($o = n/2$), the number of chunks roughly doubles. This is the key cost of overlap: it increases storage and embedding compute proportionally.

Compare the end of each chunk with the beginning of the next. You should see shared text that bridges the boundary. This keeps any information near the cut point is captured in at least one chunk's embedding. This shared text is the overlap in action: it duplicates a small window of content so that the transition zone between chunks is always fully represented somewhere in the index.

Choosing the right chunk size is one of the most impactful decisions in a RAG pipeline. It involves a fundamental trade-off between precision and context, and the optimal balance depends on the nature of your data, the types of questions you ask, and the capabilities of your embedding model. Getting chunk size right can mean the difference between a system that reliably retrieves the right passage and one that returns vaguely related blocks of text.

Think of chunk size selection as choosing the magnification level on a microscope. A very high magnification gives you fine detail, but you lose the ability to see how structures relate to one another within the broader tissue. A low magnification shows you the big picture but reveals little about individual cells. The right magnification depends on what you are trying to see. Similarly, chunk size must be chosen based on the granularity of questions you expect users to ask and the granularity of information that documents contain.

Before examining the formal trade-off, it helps to understand the two failure modes at the extremes. When chunks are too small, retrieval becomes fragile: you can find the right words, but the surrounding context is missing, and the LLM cannot produce a useful answer. When chunks are too large, retrieval becomes imprecise: the embedding represents multiple topics, and the cosine similarity between query and chunk reflects a blend of relevance rather than a strong match. Understanding these failure modes helps you diagnose problems in your own system.

The precision-context trade-off is the central tension in chunk size selection. At one extreme, you could make each chunk a single sentence, maximizing precision. At the other, you could embed entire documents, maximizing context. Neither extreme works well in practice, and understanding why illuminates the core challenge.

Small chunks (100-200 tokens) provide high retrieval precision. Each chunk covers a narrow topic, so when it matches a query, the match is likely relevant. The embedding vector represents a focused semantic concept, and the cosine similarity between query and chunk is a reliable indicator of topical alignment. However, small chunks may lack sufficient context for the LLM to generate a good answer. A chunk containing "The temperature was 42°C" is useless without knowing what system or location it refers to. The LLM receives a decontextualized fact and must either hallucinate the missing context or produce an unsatisfying response.

Large chunks (500-1,000 tokens) provide rich context. The LLM receives enough surrounding information to understand and synthesize an answer. A paragraph that introduces a concept, provides evidence, and draws a conclusion gives the LLM everything it needs to produce a coherent response. But large chunks produce less precise embeddings because they cover more topics, and they may include irrelevant information that confuses the model. When a 800-token chunk discusses three related but distinct subtopics, its embedding becomes a centroid in the semantic space that lies between all three topics. A query that is specifically about one of those subtopics may find a better cosine similarity match with a more focused chunk from a completely different document.

The optimal chunk size depends on several factors:

• Query type: Factoid questions ("What is the boiling point of water?") benefit from small chunks, while analytical questions ("Explain the causes of the 2008 financial crisis") need larger chunks with more context. If your application primarily serves one type of query, you can tune chunk size accordingly. If it serves a mix, you may need to compromise or use multiple chunk sizes.
• Document type: Technical documentation with short, self-contained sections works well with smaller chunks. Narrative text where ideas develop over paragraphs needs larger chunks. A software API reference, where each function's description is independent, naturally lends itself to small chunks. A legal brief, where arguments build across paragraphs, demands larger ones.
• Embedding model capacity: Different models have different optimal input lengths. Some models are trained on short passages and perform best with 1-2 sentences, while others handle full paragraphs effectively. Feeding a paragraph-optimized model a single sentence wastes its capacity, while feeding a sentence-optimized model a full paragraph may degrade its embedding quality.

Let's compare how different chunk sizes affect the content of chunks from the same document. By viewing the same text through different chunk size lenses, we can build intuition for how this parameter shapes the granularity and coherence of the resulting pieces.

Smaller chunk sizes produce more chunks, each covering a narrower topic. With a 100-character limit, individual sentences or pairs of short sentences become the unit of retrieval, giving the system laser-like precision but very little surrounding context. Larger chunk sizes produce fewer chunks that span multiple topics: a 500-character chunk might combine the definition of supervised learning with examples of specific algorithms, creating a richer but less focused unit.

The key insight here is that neither precision nor context is a fixed quantity: both are determined by the relationship between chunk size and document structure. A 250-character chunk of a dense academic paper might be too small to convey a complete argument, while a 250-character chunk of a FAQ document might contain an entire question-answer pair. The same number means different things in different documents.

There is no universally correct size; the right choice depends on your use case and should ideally be determined through evaluation. Most practitioners start with 256-512 tokens as a reasonable default, then adjust based on observed retrieval quality. We will cover systematic evaluation in the RAG Evaluation chapter.

Many real-world documents have explicit structural markers: Markdown headers, HTML tags, LaTeX section commands, or table of contents entries. These markers are not arbitrary formatting; they represent deliberate authorial decisions about how information is organized. A section header signals a topical boundary, and the content beneath it forms a coherent unit that the author intended to be read together.

Structural chunking uses these markers to split documents along their natural boundaries. This keeps each chunk corresponds to a coherent section or subsection. This approach works particularly well for structured documents because it uses organizational signals that other chunking methods ignore. A fixed-size chunker treats a Markdown header as just another line of text, potentially grouping it with the tail end of the previous section. A structural chunker recognizes it as a boundary, keeping each section intact and associated with its heading.

Think of structural chunking as reading the document through the author's own table of contents. The author has already done the topical segmentation work; structural chunking simply follows their lead. This is especially valuable for product manuals and API references, as well as textbooks or Wikipedia articles, where authors invest significant effort in logical organization. Ignoring that organization and applying arbitrary cuts would throw away valuable editorial work.

There is an additional benefit beyond clean boundaries: the header path. By prepending the header hierarchy to each chunk, we give the embedding model information about what broader topic the chunk belongs to. "Classification" embedded in isolation could refer to library science, machine learning, biology, or many other fields. "Classification" embedded with the header path "[Machine Learning > Supervised Learning > Classification]" is unambiguously about the machine learning subfield. This disambiguation can significantly improve retrieval precision.

Notice how each chunk includes its header context (e.g., [Supervised Learning > Classification]). This metadata helps both the embedding model and the LLM understand the chunk's position within the document's hierarchy. A chunk about "Classification" without the context that it falls under "Supervised Learning" within a "Machine Learning" document would be less informative. The header path acts as a breadcrumb trail that disambiguates the content and enriches the semantic signal available to the embedding model.

By prepending this contextual path, the resulting embedding encodes what the chunk says and where it sits within the larger knowledge structure. This is one of the few free improvements available in chunking: you are not changing the content, only adding context, and that added context improves retrieval precision with essentially no downside.

The key parameters for structural chunking are:

• max_chunk_size: The maximum size for the content of each structural section. If a section exceeds this, it is further split using a fallback strategy, typically sentence-based chunking within the section. This hybrid behavior is important because some documents contain sections that vary enormously in length. A brief introductory section might be a single sentence, while a detailed methodology section might span several pages. The fallback ensures that even oversized sections are chunked coherently, with each sub-chunk still carrying the parent section's header context.

All the strategies we have discussed so far use surface-level cues: character counts, sentence boundaries, structural markers. None of them consider the meaning of the text. A paragraph break might occur in the middle of a sustained argument, or two paragraphs separated by a heading might discuss the same topic from different angles. Surface-level chunking strategies are blind to these semantic relationships.

Semantic chunking addresses this by using embeddings to detect where topics shift within a document, and placing chunk boundaries at those transition points. Rather than relying on formatting conventions that may or may not align with topical structure, semantic chunking directly measures the conceptual continuity of the text and uses that measurement to determine where to place boundaries.

Think of semantic chunking as listening to a lecture and noticing when the speaker changes subject. You do not rely on the speaker saying "next, let's talk about..." You simply feel the shift in topic from the content itself. Semantic chunking automates this intuition by measuring similarity between consecutive sentences: when similarity drops sharply, the topic is changing and a boundary should be placed.

This approach is more computationally expensive than the methods we have seen so far, because it requires embedding every sentence in the document during the chunking step itself. But for documents where topical boundaries do not align with formatting boundaries, it can produce substantially better chunks. The cost is paid at indexing time and amortized across every retrieval query that benefits from cleaner boundaries.

The core idea is elegant: if two consecutive sentences have similar embeddings, they likely discuss the same topic and should be in the same chunk. When embeddings diverge significantly between adjacent sentences, that point likely marks a topic shift and is a natural place to split.

The semantic chunking algorithm proceeds in four steps, each building naturally on the previous one.

1. Segment text into sentences using standard sentence segmentation. Sentences serve as the finest-grained unit we consider, since splitting within a sentence would break grammatical coherence.
2. Embed each sentence using a sentence embedding model. Each sentence is mapped to a dense vector that captures its semantic content. This gives us a sequence of vectors, one per sentence, representing the "meaning trajectory" of the document.
3. Compute similarity between consecutive sentences using cosine similarity. For sentences $i$ and $i+1$ with embedding vectors $\mathbf{e}_i$ and $\mathbf{e}_{i+1}$, the similarity is:

where:

• $s_i$: the cosine similarity between sentences $i$ and $i+1$, a scalar in $[-1, 1]$
• $\mathbf{e}_i \cdot \mathbf{e}_{i+1}$: the dot product of the two embedding vectors, measuring how aligned they are in semantic space
• $\|\mathbf{e}_i\|$ and $\|\mathbf{e}_{i+1}\|$: the L2 norms of each embedding, used to normalize for vector magnitude

A value near 1.0 means the two sentences discuss nearly identical content. A value near 0.0 means they discuss unrelated topics. Values below 0.3 typically signal a significant topic shift.

1. Identify breakpoints where similarity drops below a threshold. Specifically, we compute a threshold $\tau$ at the $p$-th percentile of all similarity values:

where:

• $\tau$: the similarity threshold below which a boundary is placed
• $N$: the total number of sentences in the document
• $p$: the percentile value, typically between 10 and 40

A boundary is placed between sentences $i$ and $i+1$ whenever $s_i < \tau$. The text between consecutive boundaries forms a single chunk. The percentile-based threshold is adaptive: documents with overall high inter-sentence coherence will have a higher threshold than documents with naturally varied topics. This keeps the algorithm calibrates to the document's own baseline rather than applying a fixed global standard.

The semantic chunker detects topic transitions, placing boundaries between the astronomy, biology, and finance sections. Unlike fixed-size chunking, it produces chunks of varying length, each covering a coherent topic. Notice in particular that the document deliberately returns to astronomy in its final paragraph after the finance section. A structural chunker relying on paragraph breaks alone would separate all four paragraphs into distinct chunks without recognizing the thematic relationship between the first and last paragraphs. The semantic chunker, by contrast, detects the low similarity at the topic transitions, regardless of how the document is formatted.

The threshold parameter controls how aggressively the chunker splits. It determines the sensitivity of the algorithm to topic transitions: a lenient threshold ignores all but the most dramatic shifts, while a strict threshold reacts to even subtle changes in subject matter. A lower percentile (e.g., 10th) means only the most dramatic topic shifts trigger splits, producing larger, fewer chunks. A higher percentile (e.g., 50th) creates more, smaller chunks at every moderate topic change.

There are three natural ways to set this threshold, each with different adaptivity properties:

• Percentile-based thresholds (as above): Split at the $p$-th percentile of similarity scores. This adapts to the document's overall coherence. A document where all consecutive sentences are highly related will have a higher threshold than one with naturally varied topics. This adaptivity is the key advantage: the algorithm calibrates itself to the document's baseline level of topical coherence, only splitting where coherence drops significantly relative to the document's own norm.
• Absolute thresholds: Split whenever similarity drops below a fixed value (e.g., 0.3). This is simpler but does not adapt to documents with generally high or low inter-sentence similarity. A technical paper with consistently high inter-sentence similarity might never trigger splits with an absolute threshold of 0.3, while a casual blog post with naturally varied topics might be split into tiny fragments.
• Standard deviation-based: Split when similarity drops more than one standard deviation below the mean. This captures statistically significant topic shifts. The logic mirrors outlier detection: if most sentence pairs have similarity around 0.7 with a standard deviation of 0.1, a pair with similarity 0.5 represents a meaningful departure from the norm and likely signals a topic transition.

The key parameters for the semantic chunking implementation are:

• threshold_percentile: Controls the sensitivity of split detection. Lower values (e.g., 10) trigger fewer splits, producing large chunks that may span loosely related subtopics. Higher values (e.g., 50) trigger more splits, producing smaller chunks that isolate finer-grained topics at the risk of fragmenting coherent discussions.
• min_chunk_size: Minimum number of sentences per chunk. Prevents creating tiny fragments from transient topic shifts. Without this constraint, a single transitional sentence that happens to differ from both its neighbors could be isolated into its own chunk, producing a fragment too small to be useful for retrieval or generation.
• model_name: The SentenceTransformer model used for embedding. Models with better semantic understanding produce more accurate boundaries. The choice of model matters because the similarity scores that drive splitting are only as good as the embeddings that produce them. A model with weak topical discrimination will produce noisy similarity curves that lead to poorly placed boundaries.

To make the mechanics of each strategy concrete, let's walk through a single short passage and trace exactly what each method produces. Consider this five-sentence passage about climate science:

"The global mean surface temperature has increased by approximately 1.1 degrees Celsius since the pre-industrial period. This warming is primarily driven by increased concentrations of greenhouse gases. Carbon dioxide levels have risen from 280 parts per million to over 420 parts per million since industrialization. Methane is the second most important greenhouse gas, with concentrations more than doubling since 1750. These changes are causing widespread disruptions to weather patterns, sea levels, and ecosystems worldwide."

We will call the five sentences S1 through S5 for reference. The total character count is approximately 570 characters. Now let's trace through each method.

Character-based chunking (chunk_size=200, overlap=0): The algorithm ignores sentence boundaries entirely. Chunk 0 covers characters 0-199, which falls mid-sentence in S2. Chunk 1 covers 200-399, starting mid-sentence and ending mid-sentence. Chunk 2 covers 400-569 (the remainder). The word "Methane" is cut into "Meth" at the end of one chunk and "ane is..." at the start of the next, and neither half carries the full meaning of the sentence. Result: 3 chunks, each covering incomplete thoughts.

Token-based chunking (chunk_size=40, overlap=0): The total passage contains roughly 90 tokens. The algorithm produces approximately 3 chunks of 40, 40, and 10 tokens respectively. Each chunk is well within the embedding model's context window, and token boundaries are respected, but sentence boundaries are not. S2 will likely be split between chunk 0 and chunk 1. Result: 3 chunks with predictable token counts but arbitrary sentence splits.

Sentence-based chunking (max_chunk_size=200): The algorithm accumulates sentences until the next sentence would push the chunk over 200 characters. S1 alone is 97 characters; adding S2 (73 characters) gives 170, within the limit. Adding S3 (80 characters) would give 250, over the limit. So chunk 0 = S1 + S2. We restart: chunk 1 starts with S3, and adding S4 (73 characters) gives 153, within the limit. Adding S5 (90 characters) would give 243, over the limit. So chunk 1 = S3 + S4. Chunk 2 = S5 alone. Result: 3 chunks, each containing complete sentences and focused on a coherent sub-topic (warming trend, greenhouse gas concentrations, consequences).

Recursive chunking (chunk_size=300): The algorithm first tries to split on double newlines. The passage has none, so it tries single newlines. The passage has none either, so it tries ". " (sentence-ending periods). The first period after position 0 falls at the end of S1, at character 96. It checks whether accumulating S1 and S2 fits within 300 characters: 170 characters, yes. It checks S1 + S2 + S3: 250 characters, still within 300. It checks S1 + S2 + S3 + S4: 323, over the limit. So chunk 0 = S1 + S2 + S3. Chunk 1 = S4 + S5. Result: 2 chunks, each larger and richer in context than the sentence-based approach.

Semantic chunking (threshold_percentile=25): The algorithm embeds each of the 5 sentences and computes 4 pairwise similarities: sim(S1,S2), sim(S2,S3), sim(S3,S4), sim(S4,S5). Sentences about temperature change (S1), greenhouse gases (S2), CO2 specifically (S3), and methane specifically (S4) are all highly related, so their similarity scores are high. S5 discusses consequences (weather, sea levels, ecosystems), a somewhat different topic, so sim(S4,S5) is moderately lower. The 25th percentile threshold might fall just below sim(S4,S5), placing a boundary before S5. Result: chunk 0 = S1+S2+S3+S4 (all about greenhouse gas causes), chunk 1 = S5 (consequences).

This comparison illustrates that different strategies serve different purposes. Character-based chunking is fast but crude. Sentence-based chunking is reliable and respects linguistic structure. Recursive chunking produces larger, contextually richer units. Semantic chunking aligns boundaries with meaning, at the cost of requiring embeddings during chunking.

Let's compare all our strategies on the same document to see how they differ in practice. Applying each method to identical input text makes the differences concrete and allows us to observe how each strategy's assumptions about text structure manifest in the resulting chunks.

The results confirm our expectations. Fixed-size strategies produce uniform chunks but cut through text arbitrarily, while sentence-based and recursive strategies produce chunks of varying size that preserve linguistic coherence. The standard deviation of chunk sizes tells an important story: fixed-size methods have near-zero variance by construction, while linguistically aware methods trade that uniformity for meaningfulness.

The key insight from this comparison is that the strategies form a hierarchy of increasing sophistication and cost. Character-based splitting requires no external libraries and runs in microseconds. Token-based splitting adds a tokenization step. Sentence-based splitting adds a sentence segmentation step. Semantic chunking adds full sentence embedding during indexing. Each step up the hierarchy adds computational cost but also adds meaning-awareness to the resulting chunks. For most production systems, sentence-based or recursive chunking represents the best balance of quality and cost.

Raw chunks lose their document context. A chunk about "Section 3.2: Results" is much more useful to an LLM if it knows this section comes from "Annual Report 2023" by "Acme Corp." Enriching chunks with metadata improves both retrieval accuracy and the quality of generated answers. Metadata records the chunk's provenance and position, placing it back in its document context.

Think of metadata as the label on a file folder. Without the label, you cannot tell what is inside without opening it. With a label that says "2023 Q3 Revenue Analysis, Section 3, Page 12", you know exactly where it fits in the larger picture before you read a word. For retrieval systems, this contextual information serves two purposes: it helps the vector similarity search return more relevant results (because metadata can be used as filters), and it helps the LLM produce more grounded, citable responses.

Common metadata to attach to each chunk includes:

• Source document: filename, URL, or document ID
• Position: chunk index, page number, or section path
• Structural context: parent headers, preceding and following chunk IDs
• Document metadata: author, date, document type, language

This metadata becomes invaluable during the retrieval and generation stages. The chunk index lets you retrieve neighboring chunks for additional context: if a retrieved chunk does not contain quite enough information to answer a question, the system can automatically pull in the preceding or following chunks to expand the context window. The source attribution enables citation in the generated response, allowing the LLM to tell you exactly where its information came from. The character offsets make it possible to highlight the relevant passage in the original document, connecting generated answers back to their source material.

Beyond these basic fields, production systems often enrich chunks with computed properties that support more sophisticated retrieval. A density score can flag chunks that are likely to be high-information (many named entities, technical terms, or citations). A freshness timestamp enables time-filtered retrieval, so queries about recent events can preferentially return newer chunks. A language tag allows multilingual systems to restrict retrieval to chunks in the user's language. The chunking step is the right moment to compute and attach all of these properties, because this is the last time you have easy access to the full document context before individual chunks enter the index.

We will see how this metadata integrates with vector databases in the upcoming chapters on Vector Similarity Search and the HNSW Index.

Different document formats require specialized chunking approaches. The strategies we have discussed work well on well-formed prose, but many real-world documents contain structured content that breaks those assumptions entirely.

Code files present a particularly common challenge. A Python function that spans 80 lines cannot be split sensibly at the character level. The last 10 lines of a function contain variable assignments and return statements that only make sense in the context of the preceding 70 lines. The right unit for code chunking is the function or class definition, not an arbitrary size limit. If possible, parse the abstract syntax tree to identify natural boundaries. For Python, the ast module provides programmatic access to these boundaries. For languages without a Python AST library, regular expressions that match common function signatures often work well enough in practice.

Tables present a different challenge. A table is fundamentally relational: each row is meaningless without the column headers, and each cell value is meaningless without its row and column context. Splitting a table across chunks destroys this relational structure entirely. A chunk containing rows 15-20 of a table without the column headers from row 1 is essentially uninterpretable. The solution is to keep tables as single chunks, even if they exceed the normal chunk size limit, and to convert them to a text representation (like Markdown or CSV format) that preserves row-column relationships. If a table is too large for a single chunk, repeat the header row at the start of each sub-chunk.

The full list of special considerations by document type includes:

• Code files: Chunk along function or class boundaries rather than by line count. Parse the abstract syntax tree if possible to identify natural boundaries.
• Tables: Keep as single chunks even if they exceed the normal size limit. Convert to Markdown or CSV format that preserves row-column relationships.
• Conversational data: Chat logs and dialogue transcripts should be chunked by conversational turns or topic shifts, not by arbitrary size. Each chunk should contain enough turns to understand the context of the conversation.
• Legal and regulatory documents: These often have numbered sections and subsections with precise cross-references. Structural chunking that preserves section numbers and hierarchies is essential. A chunk referencing "as defined in Section 2.1(b)" is useless if you cannot trace that reference.

Despite the variety of strategies available, document chunking remains more art than science. No single strategy works optimally across all document types, query patterns, and embedding models. The fundamental challenge is that chunking decisions must be made at indexing time, before you know what questions you will ask. A chunk boundary that perfectly separates two topics for one query may split the exact passage needed for another. This temporal mismatch between when you chunk and when you retrieve is a structural property of the problem, not a bug that better algorithms can fully eliminate.

This points to a deeper limitation: the optimal chunking strategy is query-dependent, but chunking happens before any query is known. In an ideal world, you would chunk the same document differently for different queries: coarser chunks for broad analytical questions, finer chunks for specific factoid lookups, topic-aligned chunks for thematic questions. Practical systems cannot afford this luxury, so they must pick a strategy and chunk size that works acceptably across the distribution of expected queries. This is inherently a compromise, and it means that any production RAG system will have some queries it handles well and some it does not. Understanding this limitation helps you set realistic expectations and motivates investing in RAG evaluation infrastructure.

Semantic chunking, while conceptually appealing, introduces its own challenges. It requires running an embedding model over every sentence during indexing, which significantly increases preprocessing time and cost. For a corpus of one million documents, each with an average of 100 sentences, semantic chunking requires 100 million sentence embeddings before a single query can be answered. This is feasible with GPU infrastructure but impractical for small teams doing quick experiments. The quality of the splits depends heavily on the embedding model's ability to capture topic coherence, and models trained primarily on sentence similarity may not always detect document-level topic transitions accurately. Semantic chunking can also produce highly variable chunk sizes: a long section on a single topic might produce a single enormous chunk, while a passage that quickly surveys several topics might be split into tiny fragments. Managing these extremes with minimum and maximum size constraints is possible, but those constraints partially undermine the semantic purity of the approach.

Another practical limitation is the interaction between chunk size and the downstream LLM's context window. If you retrieve five chunks of 500 tokens each, you consume 2,500 tokens of the LLM's context for retrieved content alone. With larger chunk sizes or more retrieved results, you may exhaust the context window before including your query and system instructions. This creates a system-level constraint: chunk size, number of retrieved results, prompt template length, and LLM context window all interact. A naive approach that optimizes chunk size in isolation, without considering how many chunks you plan to retrieve, can produce a system where the retrieval quality is excellent but the final LLM response is degraded by an overloaded context window.

Finally, evaluation of chunking quality is inherently tied to end-to-end RAG performance. You cannot evaluate chunking in isolation because the same chunks might work well with one embedding model and poorly with another, or might retrieve perfectly but confuse a particular LLM. A chunk that is semantically clean but starts with a pronoun referencing an entity from the previous chunk ("It was first discovered...") may retrieve correctly but confuse the LLM about what "it" refers to. These cross-chunk coherence issues are invisible to chunking-level metrics and only surface in end-to-end evaluation. We will address this challenge systematically in the RAG Evaluation chapter later in this part.

Document chunking turns large documents into retrieval-friendly pieces that each capture a focused topic. The key takeaways from this chapter are:

• Fixed-size chunking (by characters or tokens) is simple and fast but ignores text structure, often cutting through sentences and ideas.
• Sentence-based chunking uses linguistic boundaries to ensure each chunk contains complete sentences, producing more coherent embeddings at the cost of variable chunk sizes.
• Recursive chunking respects document hierarchy by trying paragraph boundaries first, then falling back to finer-grained splits, combining structural awareness with size control.
• Semantic chunking uses embeddings to detect topic shifts, placing boundaries where the text's meaning changes most dramatically.
• Chunk overlap ensures information near boundaries appears in multiple chunks, reducing the risk of losing cross-boundary context.
• Chunk size involves a precision-context trade-off: smaller chunks give more precise retrieval but less context, while larger chunks provide richer context but less focused embeddings. Typical values range from 200 to 1,000 tokens.
• Metadata enrichment preserves document context (source, position, section hierarchy) that would otherwise be lost during chunking, enabling better retrieval and citation.
• Special document types (code, tables, legal text) require customized chunking strategies that respect their internal structure rather than applying generic size-based splits.

In the next chapter, we will examine the embedding models that convert these chunks into the dense vectors used for retrieval, completing the connection between chunking decisions and retrieval quality.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about document chunking strategies and their impact on retrieval.