Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Document Chunking: Optimizing RAG Retrieval Pipelines › Comparing Chunking Strategies

Michael BrenndoerferJanuary 23, 20263 min read

Language AI HandbookAll chapters

Part 44 of 67Retrieval-Augmented Generation

Chapter 336Document Chunking: Optimizing RAG Retrieval Pipelines › Comparing Chunking Strategies

Reading section 11 of 16 · Comparing Chunking Strategies · 3 minSection 11 of 16 · 3 min

Section 11 of 16 · 3 min read

Document Chunking: Optimizing RAG Retrieval Pipelines › Comparing Chunking StrategiesOne section · Sections in this chapter11 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's compare all our strategies on the same document to see how they differ in practice. Applying each method to identical input text makes the differences concrete and allows us to observe how each strategy's assumptions about text structure manifest in the resulting chunks.

In[33]:

Code

comparison_text = long_text  # Use the longer machine learning document for clearer differentiation

# Unpack tuple from semantic_chunk
sem_chunks, _, _ = semantic_chunk(comparison_text, threshold_percentile=30)

results = {
    "Fixed (200 chars)": chunk_by_characters(comparison_text, chunk_size=200),
    "Token (50 tokens)": chunk_by_tokens(comparison_text, chunk_size=50),
    "Sentence (200 chars)": chunk_by_sentences(
        comparison_text, max_chunk_size=200
    ),
    "Recursive (250 chars)": recursive_chunk(comparison_text, chunk_size=250),
    "Semantic": sem_chunks,
}

Out[34]:

Visualization

Bar chart comparing the number of chunks produced by five strategies: fixed-character, token, sentence-aware, recursive, and semantic chunking. — Number of chunks generated by each strategy applied to the same document. Sentence-aware and recursive splitting produce the most chunks in this example, followed by fixed-character, token, and semantic chunking.

Bar chart with error bars comparing average character length and one standard deviation across five chunking strategies. — Average chunk size and size variability (error bars show one standard deviation) for each strategy. Token-based chunks have the tightest character-length distribution here, while semantic chunks are larger and more variable because they follow topic boundaries.

Out[35]:

Console

Fixed (200 chars):
  Chunks: 9, Avg size: 184, Std: 46 chars
  First chunk preview: 'Machine learning is a subset of artificial intelligence that...
'
Token (50 tokens):
  Chunks: 6, Avg size: 276, Std: 21 chars
  First chunk preview: 'Machine learning is a subset of artificial intelligence that...
'
Sentence (200 chars):
  Chunks: 10, Avg size: 164, Std: 28 chars
  First chunk preview: 'Machine learning is a subset of artificial intelligence that...
'
Recursive (250 chars):
  Chunks: 10, Avg size: 164, Std: 43 chars
  First chunk preview: 'Machine learning is a subset of artificial intelligence that...
'
Semantic:
  Chunks: 5, Avg size: 329, Std: 154 chars
  First chunk preview: 'Machine learning is a subset of artificial intelligence that...
'

The results confirm our expectations. Fixed-size strategies produce uniform chunks but cut through text arbitrarily, while sentence-based and recursive strategies produce chunks of varying size that preserve linguistic coherence. The standard deviation of chunk sizes tells an important story: fixed-size methods have near-zero variance by construction, while linguistically aware methods trade that uniformity for meaningfulness.

The key insight from this comparison is that the strategies form a hierarchy of increasing sophistication and cost. Character-based splitting requires no external libraries and runs in microseconds. Token-based splitting adds a tokenization step. Sentence-based splitting adds a sentence segmentation step. Semantic chunking adds full sentence embedding during indexing. Each step up the hierarchy adds computational cost but also adds meaning-awareness to the resulting chunks. For most production systems, sentence-based or recursive chunking represents the best balance of quality and cost.