Document Chunking: Optimizing RAG Retrieval Pipelines › Worked Example: Chunking a Real Document

To make the mechanics of each strategy concrete, let's walk through a single short passage and trace exactly what each method produces. Consider this five-sentence passage about climate science:

"The global mean surface temperature has increased by approximately 1.1 degrees Celsius since the pre-industrial period. This warming is primarily driven by increased concentrations of greenhouse gases. Carbon dioxide levels have risen from 280 parts per million to over 420 parts per million since industrialization. Methane is the second most important greenhouse gas, with concentrations more than doubling since 1750. These changes are causing widespread disruptions to weather patterns, sea levels, and ecosystems worldwide."

We will call the five sentences S1 through S5 for reference. The total character count is approximately 570 characters. Now let's trace through each method.

Character-based chunking (chunk_size=200, overlap=0): The algorithm ignores sentence boundaries entirely. Chunk 0 covers characters 0-199, which falls mid-sentence in S2. Chunk 1 covers 200-399, starting mid-sentence and ending mid-sentence. Chunk 2 covers 400-569 (the remainder). The word "Methane" is cut into "Meth" at the end of one chunk and "ane is..." at the start of the next, and neither half carries the full meaning of the sentence. Result: 3 chunks, each covering incomplete thoughts.

Token-based chunking (chunk_size=40, overlap=0): The total passage contains roughly 90 tokens. The algorithm produces approximately 3 chunks of 40, 40, and 10 tokens respectively. Each chunk is well within the embedding model's context window, and token boundaries are respected, but sentence boundaries are not. S2 will likely be split between chunk 0 and chunk 1. Result: 3 chunks with predictable token counts but arbitrary sentence splits.

Sentence-based chunking (max_chunk_size=200): The algorithm accumulates sentences until the next sentence would push the chunk over 200 characters. S1 alone is 97 characters; adding S2 (73 characters) gives 170, within the limit. Adding S3 (80 characters) would give 250, over the limit. So chunk 0 = S1 + S2. We restart: chunk 1 starts with S3, and adding S4 (73 characters) gives 153, within the limit. Adding S5 (90 characters) would give 243, over the limit. So chunk 1 = S3 + S4. Chunk 2 = S5 alone. Result: 3 chunks, each containing complete sentences and focused on a coherent sub-topic (warming trend, greenhouse gas concentrations, consequences).

Recursive chunking (chunk_size=300): The algorithm first tries to split on double newlines. The passage has none, so it tries single newlines. The passage has none either, so it tries ". " (sentence-ending periods). The first period after position 0 falls at the end of S1, at character 96. It checks whether accumulating S1 and S2 fits within 300 characters: 170 characters, yes. It checks S1 + S2 + S3: 250 characters, still within 300. It checks S1 + S2 + S3 + S4: 323, over the limit. So chunk 0 = S1 + S2 + S3. Chunk 1 = S4 + S5. Result: 2 chunks, each larger and richer in context than the sentence-based approach.

Semantic chunking (threshold_percentile=25): The algorithm embeds each of the 5 sentences and computes 4 pairwise similarities: sim(S1,S2), sim(S2,S3), sim(S3,S4), sim(S4,S5). Sentences about temperature change (S1), greenhouse gases (S2), CO2 specifically (S3), and methane specifically (S4) are all highly related, so their similarity scores are high. S5 discusses consequences (weather, sea levels, ecosystems), a somewhat different topic, so sim(S4,S5) is moderately lower. The 25th percentile threshold might fall just below sim(S4,S5), placing a boundary before S5. Result: chunk 0 = S1+S2+S3+S4 (all about greenhouse gas causes), chunk 1 = S5 (consequences).

This comparison illustrates that different strategies serve different purposes. Character-based chunking is fast but crude. Sentence-based chunking is reliable and respects linguistic structure. Recursive chunking produces larger, contextually richer units. Semantic chunking aligns boundaries with meaning, at the cost of requiring embeddings during chunking.