Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Document Chunking: Optimizing RAG Retrieval Pipelines › Chunk Overlap

Michael BrenndoerferJanuary 23, 20265 min read

Language AI HandbookAll chapters

Part 44 of 67Retrieval-Augmented Generation

Chapter 336Document Chunking: Optimizing RAG Retrieval Pipelines › Chunk Overlap

Reading section 6 of 16 · Chunk Overlap · 5 minSection 6 of 16 · 5 min

Section 6 of 16 · 5 min read

Document Chunking: Optimizing RAG Retrieval Pipelines › Chunk OverlapOne section · Sections in this chapter6 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When you split a document into non-overlapping chunks, information that spans a chunk boundary gets split across two chunks. A key fact might have its context in one chunk and its conclusion in the next. Neither chunk alone captures the complete idea, and retrieval may miss it entirely.

Consider a passage where one sentence introduces a concept and the very next sentence provides the critical detail you are searching for. If the boundary falls between these two sentences, the first chunk contains a setup without a payoff, and the second chunk contains an answer without its question. An embedding of either chunk alone may fail to match your query. Think of a jigsaw puzzle where the most important piece of the picture straddles two boxes: you cannot see what it depicts by looking at either box alone.

Chunk overlap solves this by including some text from the end of each chunk at the beginning of the next. If your chunk size is 200 tokens and your overlap is 50 tokens, then the last 50 tokens of chunk $k$ appear again as the first 50 tokens of chunk $k+1$ . This means any passage of 50 or fewer tokens near the boundary is guaranteed to appear in full within at least one chunk.

More formally, for a document with $N$ total tokens, a chunk size of $n$ , and an overlap of $o$ , chunk $k$ covers token positions:

C_k = \text{tokens}\left[k(n - o) \;:\; k(n - o) + n\right]

where:

$k$ : the zero-indexed chunk number
$n$ : the number of tokens per chunk
$o$ : the number of overlapping tokens between adjacent chunks
$\text{tokens}[a:b]$ : the token subsequence from position $a$ to $b$

The total number of chunks produced is approximately:

K \approx \left\lceil \frac{N - o}{n - o} \right\rceil

where:

$N$ : total number of tokens in the document
$K$ : total number of chunks
$\lceil \cdot \rceil$ : the ceiling function (rounding up to the nearest integer)

Notice that increasing $o$ increases $K$ . With zero overlap, $K = \lceil N/n \rceil$ . With overlap equal to half the chunk size ( $o = n/2$ ), the number of chunks roughly doubles. This is the key cost of overlap: it increases storage and embedding compute proportionally.

Out[17]:

Visualization

Diagram showing three text chunks with overlapping regions highlighted in red. — Schematic of the chunk overlap mechanism. Three sequential chunks (blue, orange, and green) cover the document, with red shaded regions highlighting duplicated text at boundaries. Each overlap region ensures that information near a split point appears in full in at least one chunk's embedding.

The trade-off with overlap is straightforward:

More overlap means better boundary coverage, but it increases the total number of chunks and storage requirements. It also means the same text appears in multiple embeddings, which can inflate retrieval results with near-duplicate chunks. If your query matches the overlapping region, both adjacent chunks will score highly, potentially consuming two of your top- $k$ retrieval slots with content that is largely redundant.
Less overlap reduces redundancy, but risks losing context at boundaries. With zero overlap, any information that depends on content from both sides of a boundary is effectively invisible to retrieval.

A common rule of thumb is to set overlap to 10-20% of the chunk size. For a 500-token chunk, an overlap of 50-100 tokens usually provides sufficient boundary coverage without excessive duplication. This range represents a pragmatic balance: enough overlap to catch most boundary-spanning passages, but not so much that your index becomes bloated with near-identical chunks.

In practice, if you find that retrieval frequently returns two highly similar chunks from adjacent positions in the same document, you may be using too much overlap. If you find that relevant answers are being missed because critical context lands on the wrong side of a boundary, you may need more. The right value is empirical, not theoretical.

Let's see overlap in action with our token-based chunker:

In[18]:

Code

overlap_chunks = chunk_by_tokens(sample_text, chunk_size=40, overlap=10)

Out[19]:

Console

Chunk 0: 'The Amazon rainforest produces about 20 percent of the world's oxygen. It spans across nine countries in South America. The forest is home to approximately 10 percent of all species on Earth. Def'

Chunk 1: ' 10 percent of all species on Earth. Deforestation has reduced its area significantly over the past decades. Scientists estimate that about 17 percent of the Amazon has been destroyed in the last 50 years'

Chunk 2: ' Amazon has been destroyed in the last 50 years. Conservation efforts are critical to preserving this vital ecosystem. Many indigenous communities depend on the rainforest for their livelihoods. The Amazon River, which flows'

Chunk 3: ' their livelihoods. The Amazon River, which flows through the forest, is the largest river by volume in the world.'

Compare the end of each chunk with the beginning of the next. You should see shared text that bridges the boundary. This keeps any information near the cut point is captured in at least one chunk's embedding. This shared text is the overlap in action: it duplicates a small window of content so that the transition zone between chunks is always fully represented somewhere in the index.