Long-Form Generation › Hierarchical Generation
Language AI HandbookAll chapters Long-Form Generation › Hierarchical GenerationOne section · Sections in this chapter4 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Hierarchical generation generalizes the outline-based approach by decomposing the generation task into nested levels of abstraction. Rather than a flat outline followed by flat content, hierarchical generation operates at multiple granularities simultaneously. A document can be represented as a tree of decisions rather than a sequence of paragraphs: high-level choices (what to cover) constrain mid-level choices (how to structure each section), which constrain low-level choices (what each paragraph says).
The Hierarchical Decomposition Idea
Consider generating a 10,000-word technical report. A hierarchical approach might decompose this as follows. At the highest level, the model generates a document-level plan: what the report covers and its main claims. At the next level, it generates section-level plans: what each of the six sections contains and argues. At the paragraph level, it generates summaries of each paragraph's content. Only at the lowest level does it generate actual prose.
This top-down decomposition ensures that decisions made at higher levels constrain and guide decisions at lower levels. A section-level plan that says "this section demonstrates that method A outperforms method B on three benchmarks" makes it very clear what the paragraphs within that section need to accomplish. The model does not need to figure out the purpose of each paragraph from scratch; that decision was already made at a higher level.
The decomposition also has a nice property at generation time: each generation call has a clear, bounded scope. Rather than asking "write me a 10,000-word report on X," the system asks a sequence of narrow questions: "given the document purpose, what sections should it have?" then "given this section's purpose, what subsections?" then "given this subsection, what paragraphs?" then "given this paragraph's purpose, write the prose." Each individual call is well within the model's comfortable competence.
Implementation: A Three-Level Hierarchy
We can implement a simple three-level hierarchy in plain Python: document plan, section summaries, and final prose. The DocumentPlan class stores the hierarchical structure and can serialize itself as context for any generation call:
from dataclasses import dataclass, field
@dataclass
class DocumentPlan:
"""Represents the hierarchical plan for a long-form document."""
topic: str
thesis: str
sections: list = field(default_factory=list)
def add_section(self, title: str, purpose: str, key_points: list):
self.sections.append(
{
"title": title,
"purpose": purpose,
"key_points": key_points,
"paragraphs": [],
}
)
def add_paragraph_plan(self, section_idx: int, paragraph_summary: str):
self.sections[section_idx]["paragraphs"].append(paragraph_summary)
def to_prompt_context(self) -> str:
"""Format the document plan as a prompt-ready context string."""
lines = [
f"Document Topic: {self.topic}",
f"Main Thesis: {self.thesis}",
"",
]
for i, section in enumerate(self.sections):
lines.append(f"Section {i + 1}: {section['title']}")
lines.append(f" Purpose: {section['purpose']}")
lines.append(f" Key points: {', '.join(section['key_points'])}")
return "\n".join(lines)Document Plan (as context for generation): ================================================== Document Topic: Transformer Architecture for NLP Main Thesis: Transformers outperform RNNs by replacing sequential computation with parallel attention. Section 1: From RNNs to Transformers Purpose: Motivate why attention was needed Key points: RNN vanishing gradient, sequential bottleneck, attention as solution Section 2: The Attention Mechanism Purpose: Explain how self-attention works Key points: queries, keys, values, scaled dot-product, multi-head attention Section 3: Training and Results Purpose: Show empirical benefits Key points: BLEU score improvements, training speed, scalability
The to_prompt_context() method produces a compact representation of the full document plan that can be prepended to any generation prompt without consuming excessive tokens. A three-section plan like this uses roughly 80 tokens, leaving ample budget for section content. Even a 20-section plan would use at most 400-500 tokens, making it practical to include the full plan in every generation call.
Paragraph-Level Planning
Once section-level plans exist, each section can be further decomposed into paragraph plans before any prose is generated. The generation call for paragraph planning is similar to outline generation, but narrower in scope:
def generate_paragraph_plans(
document_plan: DocumentPlan, section_idx: int, num_paragraphs: int = 4
) -> list:
"""Generate paragraph-level summaries for a section."""
section = document_plan.sections[section_idx]
prompt = f"""Given this document plan:
{document_plan.to_prompt_context()}
For Section {section_idx + 1} ({section["title"]}):
Purpose: {section["purpose"]}
Key points to cover: {", ".join(section["key_points"])}
Generate {num_paragraphs} paragraph-level summaries. Each summary should be
1-2 sentences describing what that specific paragraph will argue or explain.
The paragraphs should flow logically and collectively fulfill the section's purpose."""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0.7,
max_tokens=400,
)
# Parse response into individual paragraph summaries
content = response.choices[0].message.content
paragraphs = [p.strip() for p in content.split("\n") if p.strip()]
return paragraphs[:num_paragraphs]Paragraph plans for 'From RNNs to Transformers': 1. Introduce the key limitation of RNNs: sequential computation prevents parallelization and creates vanishing gradient problems for long sequences. 2. Explain how the attention mechanism was first proposed as a solution to the information bottleneck in encoder-decoder models (Bahdanau et al., 2015). 3. Describe the conceptual leap of 'Attention is All You Need': removing recurrence entirely and building a model entirely from attention layers. 4. Summarize the performance advantages observed in the original paper, setting up the technical deep dive in the next section.
With paragraph plans in hand, generating each paragraph becomes a tightly scoped task. The model knows the exact purpose of each paragraph, what came before, and what needs to come next. This is similar to how an actor performs better when they understand their character's motivation in a scene, rather than just reading lines in isolation.
The paragraph planning step also catches structural problems early. If four planned paragraphs do not collectively fulfill the section's stated purpose, that is visible at the planning stage, before any prose has been written. Revising a four-sentence paragraph plan is far less expensive than revising four fully-written paragraphs.