Document Chunking: Optimizing RAG Retrieval Pipelines › Structural Chunking
Language AI HandbookAll chapters Document Chunking: Optimizing RAG Retrieval Pipelines › Structural ChunkingOne section · Sections in this chapter8 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Many real-world documents have explicit structural markers: Markdown headers, HTML tags, LaTeX section commands, or table of contents entries. These markers are not arbitrary formatting; they represent deliberate authorial decisions about how information is organized. A section header signals a topical boundary, and the content beneath it forms a coherent unit that the author intended to be read together.
Structural chunking uses these markers to split documents along their natural boundaries. This keeps each chunk corresponds to a coherent section or subsection. This approach works particularly well for structured documents because it uses organizational signals that other chunking methods ignore. A fixed-size chunker treats a Markdown header as just another line of text, potentially grouping it with the tail end of the previous section. A structural chunker recognizes it as a boundary, keeping each section intact and associated with its heading.
Think of structural chunking as reading the document through the author's own table of contents. The author has already done the topical segmentation work; structural chunking simply follows their lead. This is especially valuable for product manuals and API references, as well as textbooks or Wikipedia articles, where authors invest significant effort in logical organization. Ignoring that organization and applying arbitrary cuts would throw away valuable editorial work.
There is an additional benefit beyond clean boundaries: the header path. By prepending the header hierarchy to each chunk, we give the embedding model information about what broader topic the chunk belongs to. "Classification" embedded in isolation could refer to library science, machine learning, biology, or many other fields. "Classification" embedded with the header path "[Machine Learning > Supervised Learning > Classification]" is unambiguously about the machine learning subfield. This disambiguation can significantly improve retrieval precision.
import re
def chunk_by_markdown_headers(text, max_chunk_size=500):
"""Split a Markdown document by headers, preserving hierarchy."""
# Split on lines that start with one or more # characters
header_pattern = re.compile(r"^(#{1,6})\s+(.+)$", re.MULTILINE)
sections = []
last_end = 0
headers_stack = []
for match in header_pattern.finditer(text):
# Save content before this header
if last_end < match.start():
content = text[last_end : match.start()].strip()
if content and headers_stack:
sections.append(
{"header": " > ".join(headers_stack), "content": content}
)
level = len(match.group(1))
title = match.group(2)
# Update header stack based on level
while headers_stack and len(headers_stack) >= level:
headers_stack.pop()
headers_stack.append(title)
last_end = match.end()
# Capture remaining content
remaining = text[last_end:].strip()
if remaining and headers_stack:
sections.append(
{"header": " > ".join(headers_stack), "content": remaining}
)
# Combine header context with content into chunks
chunks = []
for section in sections:
chunk_text = f"[{section['header']}]\n{section['content']}"
if len(chunk_text) <= max_chunk_size:
chunks.append(chunk_text)
else:
# Fall back to sentence-level splitting within the section
sub_chunks = chunk_by_sentences(
section["content"], max_chunk_size=max_chunk_size
)
for sc in sub_chunks:
chunks.append(f"[{section['header']}]\n{sc}")
return chunksmarkdown_doc = """# Machine Learning
Machine learning enables computers to learn from data without explicit programming.
## Supervised Learning
### Classification
Classification assigns inputs to discrete categories. Common algorithms include
logistic regression, support vector machines, and neural networks. The model learns
a decision boundary that separates different classes in the feature space.
### Regression
Regression predicts continuous values. Linear regression fits a straight line to
the data, while polynomial regression can capture nonlinear relationships. Neural
networks can learn arbitrarily complex regression functions.
## Unsupervised Learning
### Clustering
Clustering groups similar data points without labels. K-means is the most popular
clustering algorithm. It partitions data into K groups by minimizing within-cluster
variance. DBSCAN is an alternative that can find clusters of arbitrary shape.
### Dimensionality Reduction
PCA projects high-dimensional data onto its principal components. t-SNE and UMAP
create nonlinear 2D projections for visualization. Autoencoders learn compressed
representations through neural networks.
"""md_chunks = chunk_by_markdown_headers(markdown_doc, max_chunk_size=400)Chunk 0: [Machine Learning] Machine learning enables computers to learn from data without explicit programming. ------------------------------ Chunk 1: [Machine Learning > Supervised Learning > Classification] Classification assigns inputs to discrete categories. Common algorithms include logistic regression, support vector machines, and neural networks. The model learns a decision boundary that separates different classes in the feature space. ------------------------------ Chunk 2: [Machine Learning > Supervised Learning > Regression] Regression predicts continuous values. Linear regression fits a straight line to the data, while polynomial regression can capture nonlinear relationships. Neural networks can learn arbitrarily complex regression functions. ------------------------------ Chunk 3: [Machine Learning > Unsupervised Learning > Clustering] Clustering groups similar data points without labels. K-means is the most popular clustering algorithm. It partitions data into K groups by minimizing within-cluster variance. DBSCAN is an alternative that can find clusters of arbitrary shape. ------------------------------ Chunk 4: [Machine Learning > Unsupervised Learning > Dimensionality Reduction] PCA projects high-dimensional data onto its principal components. t-SNE and UMAP create nonlinear 2D projections for visualization. Autoencoders learn compressed representations through neural networks. ------------------------------
Notice how each chunk includes its header context (e.g., [Supervised Learning > Classification]). This metadata helps both the embedding model and the LLM understand the chunk's position within the document's hierarchy. A chunk about "Classification" without the context that it falls under "Supervised Learning" within a "Machine Learning" document would be less informative. The header path acts as a breadcrumb trail that disambiguates the content and enriches the semantic signal available to the embedding model.
By prepending this contextual path, the resulting embedding encodes what the chunk says and where it sits within the larger knowledge structure. This is one of the few free improvements available in chunking: you are not changing the content, only adding context, and that added context improves retrieval precision with essentially no downside.
Key Parameters
The key parameters for structural chunking are:
- max_chunk_size: The maximum size for the content of each structural section. If a section exceeds this, it is further split using a fallback strategy, typically sentence-based chunking within the section. This hybrid behavior is important because some documents contain sections that vary enormously in length. A brief introductory section might be a single sentence, while a detailed methodology section might span several pages. The fallback ensures that even oversized sections are chunked coherently, with each sub-chunk still carrying the parent section's header context.