Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Document Chunking: Optimizing RAG Retrieval Pipelines › Fixed-Size Chunking

Michael BrenndoerferJanuary 23, 20268 min read

Language AI HandbookAll chapters

Part 44 of 67Retrieval-Augmented Generation

Chapter 336Document Chunking: Optimizing RAG Retrieval Pipelines › Fixed-Size Chunking

Reading section 3 of 16 · Fixed-Size Chunking · 8 minSection 3 of 16 · 8 min

Section 3 of 16 · 8 min read

Document Chunking: Optimizing RAG Retrieval Pipelines › Fixed-Size ChunkingOne section · Sections in this chapter3 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The simplest chunking strategy splits text into pieces of a fixed number of characters or tokens, regardless of where sentences or paragraphs begin or end. Despite its simplicity, fixed-size chunking is common in production RAG systems because it runs quickly and produces predictable boundaries that are easy to reason about. When you need to process millions of documents quickly and you want deterministic, reproducible behavior, fixed-size chunking is often the pragmatic choice.

Think of fixed-size chunking as using a cookie cutter on dough. The cutter produces uniform shapes, and you can process them quickly on an assembly line. The downside is that sometimes you cut through the chocolate chip or split the raisin, producing shapes that lack the distinctive feature you were hoping to capture. The uniformity is an asset for engineering; it is a liability for meaning.

Its uniformity also simplifies downstream engineering: every chunk consumes roughly the same amount of storage, embedding compute, and retrieval bandwidth. You can predict exactly how many chunks a document will produce, how much index storage you need, and how long embedding will take. For large-scale systems where these properties matter, this predictability is valuable. Many production systems use fixed-size chunking not because it is optimal, but because it is reliable enough and far easier to operate at scale than semantically aware alternatives.

Before diving into implementations, it is worth developing intuition for what "fixed-size" means in practice. Two natural choices are character counts and token counts, and they behave quite differently.

Character-Based SplittingLink Copied

The most basic approach splits text every $n$ characters. This produces chunks of uniform length but can cut words and sentences in the middle.

The mechanism is straightforward: we define a window of size $n$ and slide it along the text, recording each window as a chunk. When overlap is specified, adjacent chunks share some content. Formally, if we denote the full text as a sequence of characters $T = t_1, t_2, \ldots, t_L$ , and we set chunk size $n$ and overlap $o$ , then chunk $k$ contains characters at positions:

C_k = T[k(n-o) : k(n-o) + n]

where:

$k$ : the zero-indexed chunk number
$n$ : the chunk size in characters
$o$ : the overlap in characters (between 0 and $n$ )
$T[a:b]$ : the substring of $T$ from position $a$ to $b$

The stride between consecutive chunks is $n - o$ characters. With zero overlap, $o = 0$ , and chunks are perfectly non-overlapping; each position in the text appears in exactly one chunk. With positive overlap, positions in the overlap zone appear in two consecutive chunks.

In[3]:

Code

!uv pip install tiktoken spacy sentence-transformers matplotlib numpy
!uv pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl

def chunk_by_characters(text, chunk_size=200, overlap=0):
    """Split text into fixed-size character chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

sample_text = (
    "The Amazon rainforest produces about 20 percent of the world's oxygen. "
    "It spans across nine countries in South America. The forest is home to "
    "approximately 10 percent of all species on Earth. Deforestation has reduced "
    "its area significantly over the past decades. Scientists estimate that about "
    "17 percent of the Amazon has been destroyed in the last 50 years. Conservation "
    "efforts are critical to preserving this vital ecosystem. Many indigenous "
    "communities depend on the rainforest for their livelihoods. The Amazon River, "
    "which flows through the forest, is the largest river by volume in the world."
)

In[4]:

Code

chunks = chunk_by_characters(sample_text, chunk_size=150)

Out[5]:

Console

Chunk 0: [150 chars] 'The Amazon rainforest produces about 20 percent of the world's oxygen. It spans across nine countries in South America. The forest is home to approxim'
Chunk 1: [150 chars] 'ately 10 percent of all species on Earth. Deforestation has reduced its area significantly over the past decades. Scientists estimate that about 17 pe'
Chunk 2: [150 chars] 'rcent of the Amazon has been destroyed in the last 50 years. Conservation efforts are critical to preserving this vital ecosystem. Many indigenous com'
Chunk 3: [150 chars] 'munities depend on the rainforest for their livelihoods. The Amazon River, which flows through the forest, is the largest river by volume in the world'
Chunk 4: [1 chars] '.'

Notice how chunks cut through words and sentences without regard for meaning. Chunk 1 might start in the middle of a word, making it difficult for an embedding model to capture the intended meaning. A phrase like "approximately 10 percent of all species" could be severed into "approxi" at the end of one chunk and "mately 10 percent of all species" at the beginning of the next, rendering both fragments less semantically coherent than the original sentence.

This is the fundamental weakness of character-based splitting: it optimizes for uniform size at the expense of coherence. The embedding model must then try to extract meaning from text that may begin or end mid-thought, producing embeddings that are noisier and less representative of any single idea. For use cases where speed and simplicity matter more than embedding quality, character-based splitting is a reasonable starting point. For use cases where retrieval precision is critical, it almost always warrants improvement.

Token-Based SplittingLink Copied

A better variant splits by token count rather than character count. Since embedding models operate on tokens, not raw characters, this ensures each chunk uses the model's capacity efficiently. A character-based chunk of 200 characters might translate to anywhere from 30 to 60 tokens depending on word length and vocabulary, creating unpredictable utilization of the embedding model's context window. Token-based splitting eliminates this variability by directly controlling the unit that matters.

The key insight here is alignment: you want to control the input size in the same units the model uses internally. Controlling characters while the model measures tokens is like setting a cooking timer in Fahrenheit when the recipe is in Celsius. You can make it work, but you are adding unnecessary conversion error. Token-based splitting removes that friction.

We can use the tiktoken library, which implements the tokenizers used by OpenAI models, to perform this conversion accurately. The cl100k_base encoding is the tokenizer used by GPT-4 and GPT-3.5-turbo, and it is widely used as a proxy tokenizer even for non-OpenAI models because it has well-understood behavior on English text.

In[6]:

Code

import tiktoken


def chunk_by_tokens(
    text, chunk_size=50, overlap=0, encoding_name="cl100k_base"
):
    """Split text into chunks of a fixed number of tokens."""
    enc = tiktoken.get_encoding(encoding_name)
    tokens = enc.encode(text)

    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunk_text = enc.decode(chunk_tokens)
        chunks.append(chunk_text)
        start += chunk_size - overlap
    return chunks

In[7]:

Code

token_chunks = chunk_by_tokens(sample_text, chunk_size=40, overlap=0)

Out[8]:

Console

Chunk 0: [40 tokens] 'The Amazon rainforest produces about 20 percent of the world's oxygen. It spans across nine countries in South America. The forest is home to approximately 10 percent of all species on Earth. Def'
Chunk 1: [40 tokens] 'orestation has reduced its area significantly over the past decades. Scientists estimate that about 17 percent of the Amazon has been destroyed in the last 50 years. Conservation efforts are critical to preserving this vital ecosystem'
Chunk 2: [34 tokens] '. Many indigenous communities depend on the rainforest for their livelihoods. The Amazon River, which flows through the forest, is the largest river by volume in the world.'

Out[9]:

Visualization

Histogram showing the distribution of character lengths for fixed 50-token chunks, with a red dashed vertical line marking the mean character length. — Distribution of character lengths for complete 50-token chunks, generated by repeating the sample text. Although every chunk contains exactly 50 tokens, character lengths vary because some words tokenize to more tokens than others. The red dashed line marks the mean character length.

Token-based splitting guarantees each chunk uses a predictable number of tokens, but it still does not respect sentence boundaries. A sentence split across two chunks loses coherence in both. The first chunk ends with an incomplete thought, and the second begins without the context established earlier in the sentence. For many retrieval scenarios, this partial-sentence problem degrades embedding quality enough to motivate the sentence-aware approaches we explore next.

Key ParametersLink Copied

The key parameters for fixed-size chunking are:

chunk_size: The target size of each chunk (in characters or tokens). Smaller chunks are more precise, while larger chunks provide more context. This parameter directly controls the trade-off between retrieval granularity and the amount of information each chunk carries.
overlap: The number of units (characters or tokens) repeated between adjacent chunks to prevent information loss at boundaries. Overlap acts as a safety net. This keeps content near a cut point is fully represented in at least one chunk. We explore this concept in depth in the Chunk Overlap section below.