Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Document Chunking: Optimizing RAG Retrieval Pipelines › Semantic Chunking

Michael BrenndoerferJanuary 23, 20269 min read

Language AI HandbookAll chapters

Part 44 of 67Retrieval-Augmented Generation

Chapter 336Document Chunking: Optimizing RAG Retrieval Pipelines › Semantic Chunking

Reading section 9 of 16 · Semantic Chunking · 9 minSection 9 of 16 · 9 min

Section 9 of 16 · 9 min read

Document Chunking: Optimizing RAG Retrieval Pipelines › Semantic ChunkingOne section · Sections in this chapter9 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

All the strategies we have discussed so far use surface-level cues: character counts, sentence boundaries, structural markers. None of them consider the meaning of the text. A paragraph break might occur in the middle of a sustained argument, or two paragraphs separated by a heading might discuss the same topic from different angles. Surface-level chunking strategies are blind to these semantic relationships.

Semantic chunking addresses this by using embeddings to detect where topics shift within a document, and placing chunk boundaries at those transition points. Rather than relying on formatting conventions that may or may not align with topical structure, semantic chunking directly measures the conceptual continuity of the text and uses that measurement to determine where to place boundaries.

Think of semantic chunking as listening to a lecture and noticing when the speaker changes subject. You do not rely on the speaker saying "next, let's talk about..." You simply feel the shift in topic from the content itself. Semantic chunking automates this intuition by measuring similarity between consecutive sentences: when similarity drops sharply, the topic is changing and a boundary should be placed.

This approach is more computationally expensive than the methods we have seen so far, because it requires embedding every sentence in the document during the chunking step itself. But for documents where topical boundaries do not align with formatting boundaries, it can produce substantially better chunks. The cost is paid at indexing time and amortized across every retrieval query that benefits from cleaner boundaries.

The core idea is elegant: if two consecutive sentences have similar embeddings, they likely discuss the same topic and should be in the same chunk. When embeddings diverge significantly between adjacent sentences, that point likely marks a topic shift and is a natural place to split.

AlgorithmLink Copied

The semantic chunking algorithm proceeds in four steps, each building naturally on the previous one.

Segment text into sentences using standard sentence segmentation. Sentences serve as the finest-grained unit we consider, since splitting within a sentence would break grammatical coherence.
Embed each sentence using a sentence embedding model. Each sentence is mapped to a dense vector that captures its semantic content. This gives us a sequence of vectors, one per sentence, representing the "meaning trajectory" of the document.
Compute similarity between consecutive sentences using cosine similarity. For sentences $i$ and $i+1$ with embedding vectors $\mathbf{e}_i$ and $\mathbf{e}_{i+1}$ , the similarity is:

s_i = \frac{\mathbf{e}_i \cdot \mathbf{e}_{i+1}}{\|\mathbf{e}_i\| \cdot \|\mathbf{e}_{i+1}\|}

where:

$s_i$ : the cosine similarity between sentences $i$ and $i+1$ , a scalar in $[-1, 1]$
$\mathbf{e}_i \cdot \mathbf{e}_{i+1}$ : the dot product of the two embedding vectors, measuring how aligned they are in semantic space
$\|\mathbf{e}_i\|$ and $\|\mathbf{e}_{i+1}\|$ : the L2 norms of each embedding, used to normalize for vector magnitude

A value near 1.0 means the two sentences discuss nearly identical content. A value near 0.0 means they discuss unrelated topics. Values below 0.3 typically signal a significant topic shift.

Identify breakpoints where similarity drops below a threshold. Specifically, we compute a threshold $\tau$ at the $p$ -th percentile of all similarity values:

\tau = \text{percentile}(s_1, s_2, \ldots, s_{N-1},\; p)

where:

$\tau$ : the similarity threshold below which a boundary is placed
$N$ : the total number of sentences in the document
$p$ : the percentile value, typically between 10 and 40

A boundary is placed between sentences $i$ and $i+1$ whenever $s_i < \tau$ . The text between consecutive boundaries forms a single chunk. The percentile-based threshold is adaptive: documents with overall high inter-sentence coherence will have a higher threshold than documents with naturally varied topics. This keeps the algorithm calibrates to the document's own baseline rather than applying a fixed global standard.

In[28]:

Code

import numpy as np
from sentence_transformers import SentenceTransformer

_embedding_models = {}


def semantic_chunk(
    text,
    model_name="all-MiniLM-L6-v2",
    threshold_percentile=25,
    min_chunk_size=2,
):
    """Split text into chunks based on semantic similarity between sentences."""
    # Step 1: Segment into sentences
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents if sent.text.strip()]

    if len(sentences) <= min_chunk_size:
        return [text], sentences, []

    # Step 2: Embed each sentence
    # Reuse the model when comparing several documents in one notebook run.
    if model_name not in _embedding_models:
        _embedding_models[model_name] = SentenceTransformer(model_name)
    model = _embedding_models[model_name]
    embeddings = model.encode(sentences)

    # Step 3: Compute cosine similarity between consecutive sentences
    similarities = []
    for i in range(len(embeddings) - 1):
        sim = np.dot(embeddings[i], embeddings[i + 1]) / (
            np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i + 1])
        )
        similarities.append(sim)

    # Step 4: Find breakpoints where similarity drops below threshold
    threshold = np.percentile(similarities, threshold_percentile)
    breakpoints = [
        i + 1 for i, sim in enumerate(similarities) if sim < threshold
    ]

    # Build chunks from breakpoints
    chunks = []
    start = 0
    for bp in breakpoints:
        chunk_sentences = sentences[start:bp]
        if len(chunk_sentences) >= min_chunk_size:
            chunks.append(" ".join(chunk_sentences))
            start = bp

    # Add remaining sentences
    if start < len(sentences):
        remaining = " ".join(sentences[start:])
        if chunks and len(sentences[start:]) < min_chunk_size:
            chunks[-1] += " " + remaining
        else:
            chunks.append(remaining)

    return chunks, sentences, similarities

Let's apply semantic chunking to a document that covers multiple distinct topics:

In[29]:

Code

multi_topic_text = """
The solar system consists of eight planets orbiting the Sun. Mercury is the closest
planet to the Sun and has no atmosphere. Venus is the hottest planet due to its thick
carbon dioxide atmosphere. Earth is the only known planet to support life.

Photosynthesis is the process by which plants convert sunlight into energy. Chlorophyll
in plant cells absorbs light energy. The process produces oxygen as a byproduct.
Plants use carbon dioxide and water as inputs for photosynthesis.

The stock market experienced significant volatility in 2023. Interest rate hikes by
central banks affected investor sentiment. Technology stocks showed strong recovery
in the second half of the year. Cryptocurrency markets also saw renewed interest from
institutional investors.

Returning to astronomy, the James Webb Space Telescope has revealed new details about
distant galaxies. Its infrared capabilities allow it to see through cosmic dust.
Scientists have discovered exoplanets in habitable zones using data from the telescope.
""".strip()

In[30]:

Code

result = semantic_chunk(multi_topic_text, threshold_percentile=30)
sem_chunks, sentences, similarities = result

Out[31]:

Visualization

Line chart of sentence-to-sentence cosine similarity with dips at topic transitions. — Cosine similarity scores between consecutive sentences in a multi-topic document covering astronomy, photosynthesis, finance, and space telescopes. Deep troughs in the similarity curve (below the dashed red threshold) mark topic transitions, which the semantic chunker uses as split points. Notice the similarity recovery when the text returns to astronomy in the final paragraph.

Out[32]:

Console

Number of semantic chunks: 5

Chunk 0: 'The solar system consists of eight planets orbiting the Sun. Mercury is the closest planet to the Sun and has no atmosph...
'
Chunk 1: 'Photosynthesis is the process by which plants convert sunlight into energy. Chlorophyll in plant cells absorbs light ene...
'
Chunk 2: 'The stock market experienced significant volatility in 2023. Interest rate hikes by central banks affected investor sent...
'
Chunk 3: 'Technology stocks showed strong recovery in the second half of the year. Cryptocurrency markets also saw renewed interes...
'
Chunk 4: 'Returning to astronomy, the James Webb Space Telescope has revealed new details about distant galaxies. Its infrared cap...
'

The semantic chunker detects topic transitions, placing boundaries between the astronomy, biology, and finance sections. Unlike fixed-size chunking, it produces chunks of varying length, each covering a coherent topic. Notice in particular that the document deliberately returns to astronomy in its final paragraph after the finance section. A structural chunker relying on paragraph breaks alone would separate all four paragraphs into distinct chunks without recognizing the thematic relationship between the first and last paragraphs. The semantic chunker, by contrast, detects the low similarity at the topic transitions, regardless of how the document is formatted.

Choosing the ThresholdLink Copied

The threshold parameter controls how aggressively the chunker splits. It determines the sensitivity of the algorithm to topic transitions: a lenient threshold ignores all but the most dramatic shifts, while a strict threshold reacts to even subtle changes in subject matter. A lower percentile (e.g., 10th) means only the most dramatic topic shifts trigger splits, producing larger, fewer chunks. A higher percentile (e.g., 50th) creates more, smaller chunks at every moderate topic change.

There are three natural ways to set this threshold, each with different adaptivity properties:

Percentile-based thresholds (as above): Split at the $p$ -th percentile of similarity scores. This adapts to the document's overall coherence. A document where all consecutive sentences are highly related will have a higher threshold than one with naturally varied topics. This adaptivity is the key advantage: the algorithm calibrates itself to the document's baseline level of topical coherence, only splitting where coherence drops significantly relative to the document's own norm.
Absolute thresholds: Split whenever similarity drops below a fixed value (e.g., 0.3). This is simpler but does not adapt to documents with generally high or low inter-sentence similarity. A technical paper with consistently high inter-sentence similarity might never trigger splits with an absolute threshold of 0.3, while a casual blog post with naturally varied topics might be split into tiny fragments.
Standard deviation-based: Split when similarity drops more than one standard deviation below the mean. This captures statistically significant topic shifts. The logic mirrors outlier detection: if most sentence pairs have similarity around 0.7 with a standard deviation of 0.1, a pair with similarity 0.5 represents a meaningful departure from the norm and likely signals a topic transition.

Key ParametersLink Copied

The key parameters for the semantic chunking implementation are:

threshold_percentile: Controls the sensitivity of split detection. Lower values (e.g., 10) trigger fewer splits, producing large chunks that may span loosely related subtopics. Higher values (e.g., 50) trigger more splits, producing smaller chunks that isolate finer-grained topics at the risk of fragmenting coherent discussions.
min_chunk_size: Minimum number of sentences per chunk. Prevents creating tiny fragments from transient topic shifts. Without this constraint, a single transitional sentence that happens to differ from both its neighbors could be isolated into its own chunk, producing a fragment too small to be useful for retrieval or generation.
model_name: The SentenceTransformer model used for embedding. Models with better semantic understanding produce more accurate boundaries. The choice of model matters because the similarity scores that drive splitting are only as good as the embeddings that produce them. A model with weak topical discrimination will produce noisy similarity curves that lead to poorly placed boundaries.