Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

RAG Prompt Engineering: Context Placement & Citation Strategies › Context Truncation Strategies

Michael BrenndoerferJanuary 31, 202611 min read

Language AI HandbookAll chapters

Part 44 of 67Retrieval-Augmented Generation

Chapter 344RAG Prompt Engineering: Context Placement & Citation Strategies › Context Truncation Strategies

Reading section 4 of 11 · Context Truncation Strategies · 11 minSection 4 of 11 · 11 min

Section 4 of 11 · 11 min read

RAG Prompt Engineering: Context Placement & Citation Strategies › Context Truncation StrategiesOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Even with efficient retrieval and reranking, you will frequently encounter situations where the combined retrieved documents exceed your model's context window. Context windows have expanded dramatically in recent years, but so has the volume of retrievable content. A system retrieving from a million-document corpus may surface twenty highly relevant passages, each thousands of tokens long, for a total that vastly exceeds even a 128K-token context window. When truncation is necessary, how you truncate matters immensely: a poorly truncated document that cuts off its key sentence is worse than no document at all, because it occupies space while providing no useful information.

Think of truncation as budget allocation under uncertainty. You have a fixed token budget for all retrieved content, and you must distribute it across documents in a way that maximizes the expected relevance of what you keep. Different truncation strategies reflect different assumptions about where useful information lives within a document. The inverted pyramid assumption (information density decreases from beginning to end) favors keeping the head. The conclusions-first assumption (answers live in summaries and conclusions) favors keeping the tail. The query-centric assumption (relevant information can appear anywhere, but clusters around query-similar sentences) favors sliding window extraction.

The mathematical challenge of truncation involves optimizing an information preservation function subject to a constraint on token count. Given a document $d$ consisting of tokens $t_1, t_2, \ldots, t_n$ , and a maximum token budget $k$ , we seek to extract a subsequence $d'$ that maximizes the relevance function $R(d', q)$ with respect to query $q$ , subject to the constraint $|d'| \leq k$ .

Formally, this is a constrained optimization problem:

\max_{d' \subseteq d} R(d', q) \quad \text{subject to} \quad |d'| \leq k

where:

$d'$ : the extracted subsequence of document tokens, a subset of $d$
$R(d', q)$ : the semantic relevance of the extracted passage to the query $q$ , estimated by embedding similarity or another scoring function
$k$ : the token budget, a hard constraint determined by the context window size minus the space reserved for instructions and the user query

Different truncation strategies represent different approximations of this optimization problem based on assumptions about document structure and information distribution. None of them solve the problem exactly because computing $R(d', q)$ for all possible subsequences is combinatorially intractable. Instead, each strategy uses a structural heuristic to identify a tractable approximate solution.

Head-Only TruncationLink Copied

The simplest approach keeps the beginning of each document and truncates the end:

In[5]:

Code

# Mock tokenizer for demonstration (replace with transformers tokenizer in production)
class MockTokenizer:
    def encode(self, text):
        return text.split()

    def decode(self, tokens):
        return " ".join(tokens)


tokenizer = MockTokenizer()


def truncate_head_only(documents, max_tokens_per_doc):
    truncated = []
    for doc in documents:
        tokens = tokenizer.encode(doc)
        truncated.append(tokenizer.decode(tokens[:max_tokens_per_doc]))
    return truncated

This strategy operates on the assumption that information follows an inverted pyramid structure, common in journalistic and academic writing, where the most salient content appears in the introduction. Mathematically, this assumes that the relevance function $R$ is monotonically decreasing with position index $i$ :

R(t_i, q) \geq R(t_{i+1}, q) \quad \text{for all } i \in [1, n-1]

This preserves introductions and thesis statements but loses conclusions and summaries. It works well for academic papers and news articles where the inverted pyramid structure places key information early. It fails badly for technical documentation structured as problem-solution pairs where the solution appears at the end, or for question-answer formatted content where the answer only appears after a lengthy preamble.

Head-only truncation is also computationally simple: it requires no embeddings, no sentence segmentation, and no query awareness. This makes it the default choice for many production systems despite its obvious limitations, because it is the only strategy that adds zero latency. When you are serving thousands of requests per second, even a few milliseconds of additional latency from smarter truncation can matter.

Tail-Only TruncationLink Copied

Conversely, keeping only the end of documents preserves conclusions and final summaries:

In[6]:

Code

# Mock tokenizer for demonstration (replace with transformers tokenizer in production)
class MockTokenizer:
    def encode(self, text):
        return text.split()

    def decode(self, tokens):
        return " ".join(tokens)


tokenizer = MockTokenizer()


def truncate_tail_only(documents, max_tokens_per_doc):
    truncated = []
    for doc in documents:
        tokens = tokenizer.encode(doc)
        truncated.append(tokenizer.decode(tokens[-max_tokens_per_doc:]))
    return truncated

This approach inverts the monotonicity assumption, positing that $R(t_i, q) \leq R(t_{i+1}, q)$ . This suits documents where the final paragraphs contain the synthesis or answer, such as FAQ entries, legal briefs where the conclusion of law appears at the end, or research papers where the results section follows the methodology. The key risk with tail truncation is that it often discards the context necessary to interpret the kept content. A conclusion that says "therefore, the treatment was effective" without the preceding description of what treatment was tested provides little actionable information.

Sliding Window ExtractionLink Copied

For long documents where relevant information might appear anywhere, a sliding window approach extracts segments around query-relevant sentences:

In[7]:

Code

!uv pip install scikit-learn -q

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Mock components for demonstration (replace with actual NLTK/spacy/transformers in production)
class MockSentenceSplitter:
    def split(self, text):
        return [s.strip() for s in text.split('.') if s.strip()]

class MockEmbedder:
    def encode(self, sentences):
        # Return deterministic dummy embeddings for demonstration
        np.random.seed(0)
        return np.random.randn(len(sentences), 384)

sentence_splitter = MockSentenceSplitter()
embedder = MockEmbedder()

def extract_relevant_window(document, query_embedding, window_size=512):
    sentences = sentence_splitter.split(document)
    sentence_embeddings = embedder.encode(sentences)

    # Find most similar sentence to query
    similarities = cosine_similarity(query_embedding, sentence_embeddings)
    center_idx = np.argmax(similarities)

    # Extract window around center
    start = max(0, center_idx - window_size // 2)
    end = min(len(sentences), center_idx + window_size // 2)

    return " ".join(sentences[start:end])

This approach treats the truncation problem as a center-finding optimization. Given a query embedding vector $\mathbf{q} \in \mathbb{R}^d$ and sentence embeddings $\mathbf{s}_1, \mathbf{s}_2, \ldots, \mathbf{s}_n \in \mathbb{R}^d$ , we calculate cosine similarities:

\text{sim}(\mathbf{q}, \mathbf{s}_i) = \frac{\mathbf{q} \cdot \mathbf{s}_i}{\|\mathbf{q}\| \|\mathbf{s}_i\|}

where:

$\mathbf{q}$ : the query embedding vector in $\mathbb{R}^d$
$\mathbf{s}_i$ : the embedding of the $i$ -th sentence in the document
$\|\cdot\|$ : the Euclidean norm, used to normalize vectors before computing similarity

The optimal center position $c$ is determined by:

c = \arg\max_{i} \text{sim}(\mathbf{q}, \mathbf{s}_i)

The extraction window $[c - w/2, c + w/2]$ then captures the local context around the maximally relevant sentence, under the assumption that semantic relevance clusters locally in document space. This assumption holds reasonably well for factual documents where a single section discusses any given topic, but breaks down for complex analytical documents where a single sentence might draw on themes established pages earlier.

The sliding window approach requires an additional embedding step at inference time, adding latency proportional to the number of sentences in the document. For documents of a few thousand tokens, this is typically less than 50ms with a fast embedding model. For very long documents, you may want to segment at the paragraph level rather than the sentence level to keep this computation tractable.

Out[8]:

Visualization

Full document relevance curve showing the complete distribution of query-relevant information across all 100 positions. The document exhibits higher relevance in the introduction (positions 0-20) and conclusion (positions 80-100), with a notable peak in the middle (positions 35-45) representing a key relevant section.

Area chart of document relevance with the first 30 positions highlighted in blue as retained, showing the head-only truncation strategy. — Head-only truncation strategy preserving the first 30% of the document (positions 0-30). This approach captures the introduction's high relevance but loses the middle peak and conclusion, suitable for documents following an inverted pyramid structure.

Area chart of document relevance with the last 30 positions highlighted in purple as retained, showing the tail-only truncation strategy. — Tail-only truncation strategy preserving the final 30% of the document (positions 70-100). This method retains the conclusion and build-up sections but sacrifices the introduction and middle content, appropriate for documents where synthesis appears at the end.

Area chart of document relevance with a 30-position window centered on the maximum relevance point highlighted in orange, showing the sliding window strategy. — Sliding window extraction strategy selecting a 30-position window centered on the maximum relevance point (position 40). This query-centric approach captures the 'golden nugget' in the middle while discarding both introduction and conclusion, maximizing local relevance density.

Hierarchical SummarizationLink Copied

When individual documents are too long for simple truncation, hierarchical approaches create distillations that preserve more of the semantic content. The idea is to use a fast, cheap model to compress each document before including it in the main prompt, trading some detail for coverage.

The process works in three stages. First, you summarize each long document independently using either an extractive method (selecting the most relevant sentences) or an abstractive method (generating a shorter paraphrase). Second, you use these summaries as the context instead of the full text, allowing you to include information from many more documents within the same token budget. Third, for the top-scoring documents, you may optionally include the full text alongside the summary, giving the model access to fine-grained detail where it matters most.

This approach trades granularity for coverage: you can represent twenty documents instead of five, but each representation is lossy. The most significant risk is that the summarization step introduces its own errors. A summary model that misses a key qualifier, such as "the drug was effective only in patients over 65," can lead the main model to generate a response that omits critical nuance. Summarization before retrieval, at indexing time, is generally more reliable than summarization at inference time because it can be validated and corrected before it affects users.