Document Chunking: Optimizing RAG Retrieval Pipelines › Metadata Enrichment
Language AI HandbookAll chapters Document Chunking: Optimizing RAG Retrieval Pipelines › Metadata EnrichmentOne section · Sections in this chapter12 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Raw chunks lose their document context. A chunk about "Section 3.2: Results" is much more useful to an LLM if it knows this section comes from "Annual Report 2023" by "Acme Corp." Enriching chunks with metadata improves both retrieval accuracy and the quality of generated answers. Metadata records the chunk's provenance and position, placing it back in its document context.
Think of metadata as the label on a file folder. Without the label, you cannot tell what is inside without opening it. With a label that says "2023 Q3 Revenue Analysis, Section 3, Page 12", you know exactly where it fits in the larger picture before you read a word. For retrieval systems, this contextual information serves two purposes: it helps the vector similarity search return more relevant results (because metadata can be used as filters), and it helps the LLM produce more grounded, citable responses.
Common metadata to attach to each chunk includes:
- Source document: filename, URL, or document ID
- Position: chunk index, page number, or section path
- Structural context: parent headers, preceding and following chunk IDs
- Document metadata: author, date, document type, language
from dataclasses import dataclass, field
import tiktoken
@dataclass
class Chunk:
text: str
index: int
source: str
section: str = ""
start_char: int = 0
end_char: int = 0
metadata: dict = field(default_factory=dict)
@property
def token_count(self):
enc = tiktoken.get_encoding("cl100k_base")
return len(enc.encode(self.text))
def create_enriched_chunks(text, source, chunk_size=200):
"""Create chunks with metadata."""
raw_chunks = chunk_by_sentences(text, max_chunk_size=chunk_size)
enriched = []
char_offset = 0
for i, chunk_text in enumerate(raw_chunks):
start = text.find(chunk_text, char_offset)
end = (
start + len(chunk_text)
if start >= 0
else char_offset + len(chunk_text)
)
chunk = Chunk(
text=chunk_text,
index=i,
source=source,
start_char=max(start, 0),
end_char=end,
metadata={
"total_chunks": len(raw_chunks),
"has_previous": i > 0,
"has_next": i < len(raw_chunks) - 1,
},
)
enriched.append(chunk)
char_offset = end
return enrichedenriched = create_enriched_chunks(
sample_text, source="amazon_facts.txt", chunk_size=200
)Chunk 0/3: Source: amazon_facts.txt Tokens: 39 Chars [0:191] Text: 'The Amazon rainforest produces about 20 percent of the world's oxygen. It spans ... ' Chunk 1/3: Source: amazon_facts.txt Tokens: 32 Chars [192:360] Text: 'Deforestation has reduced its area significantly over the past decades. Scientis... ' Chunk 2/3: Source: amazon_facts.txt Tokens: 24 Chars [361:506] Text: 'Conservation efforts are critical to preserving this vital ecosystem. Many indig... '
This metadata becomes invaluable during the retrieval and generation stages. The chunk index lets you retrieve neighboring chunks for additional context: if a retrieved chunk does not contain quite enough information to answer a question, the system can automatically pull in the preceding or following chunks to expand the context window. The source attribution enables citation in the generated response, allowing the LLM to tell you exactly where its information came from. The character offsets make it possible to highlight the relevant passage in the original document, connecting generated answers back to their source material.
Beyond these basic fields, production systems often enrich chunks with computed properties that support more sophisticated retrieval. A density score can flag chunks that are likely to be high-information (many named entities, technical terms, or citations). A freshness timestamp enables time-filtered retrieval, so queries about recent events can preferentially return newer chunks. A language tag allows multilingual systems to restrict retrieval to chunks in the user's language. The chunking step is the right moment to compute and attach all of these properties, because this is the last time you have easy access to the full document context before individual chunks enter the index.
We will see how this metadata integrates with vector databases in the upcoming chapters on Vector Similarity Search and the HNSW Index.