Long-Form Generation › Coherence Maintenance
Language AI HandbookAll chapters Long-Form Generation › Coherence MaintenanceOne section · Sections in this chapter5 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Even with careful planning, generated text can lose coherence. Coherence in writing operates at multiple levels, and maintaining it requires active effort during generation. Planning establishes the structure; coherence maintenance ensures the structure holds together as actual sentences and paragraphs fill in.
Types of Coherence
Writing researchers identify several types of coherence that a document must maintain simultaneously.
Entity coherence refers to consistent reference to the same entities throughout the text. If a document introduces "the transformer model" and later refers to it as "the architecture," "the system," and "the approach" interchangeably, readers can follow this. But if the model changes an entity's properties (giving a character a different job, changing a model's architecture) across sections, the text becomes confusing or internally contradictory. Entity coherence failures are among the most jarring for readers, because they directly contradict the reader's internal model of the document's world.
Relational coherence refers to logical connections between sentences and paragraphs. Each sentence should be grounded in what came before it, connected through explicit discourse relations like causation, elaboration, contrast, or exemplification. Texts that jump between unconnected statements feel choppy even if each statement is individually true. Relational coherence is largely a local phenomenon: it primarily concerns connections between adjacent or nearby sentences.
Topical coherence refers to maintaining a consistent topic focus across sections of text. A paragraph that starts discussing attention mechanisms should end on attention mechanisms, not have drifted to a discussion of training infrastructure by its final sentence. Unlike relational coherence, topical coherence is a global property, connecting the content of the current passage to the document's overall subject.
Temporal coherence is a fourth type relevant for narrative and instructional text. Events described in a story or tutorial should follow a consistent timeline, and procedural steps should appear in the order they need to be executed. Language models generating instructional text occasionally produce steps out of sequence, creating instructions that are confusing or incorrect to follow.
Entity Tracking
One practical approach to entity coherence is explicit entity tracking. The generator maintains a registry of named entities introduced in the document and their key attributes. Before generating each new section, this registry is serialized and injected into the prompt. The registry is a persistent memory that compensates for the model's inability to remember earlier content reliably.
@dataclass
class Entity:
"""Tracks a named entity and its attributes throughout a document."""
name: str
entity_type: str # person, concept, model, dataset, etc.
aliases: list = field(default_factory=list)
attributes: dict = field(default_factory=dict)
first_mentioned: Optional[int] = None # Section index
@dataclass
class EntityRegistry:
"""Maintains consistent tracking of all entities in a document."""
entities: dict = field(default_factory=dict)
def register(
self, name: str, entity_type: str, section_idx: int, **attributes
):
if name not in self.entities:
self.entities[name] = Entity(
name=name,
entity_type=entity_type,
attributes=attributes,
first_mentioned=section_idx,
)
else:
# Update attributes for existing entity
self.entities[name].attributes.update(attributes)
def add_alias(self, canonical_name: str, alias: str):
if canonical_name in self.entities:
self.entities[canonical_name].aliases.append(alias)
def to_context_string(self) -> str:
"""Format entity registry for prompt injection."""
lines = ["Key entities in this document:"]
for name, entity in self.entities.items():
attrs = ", ".join(f"{k}: {v}" for k, v in entity.attributes.items())
aliases = (
f" (also: {', '.join(entity.aliases)})"
if entity.aliases
else ""
)
lines.append(f" - {name}{aliases} [{entity.entity_type}]: {attrs}")
return "\n".join(lines)Key entities in this document: - BERT (also: the encoder model) [model]: type: encoder-only transformer, year: 2018, developers: Google Brain - GPT-3 (also: the 175-billion parameter model) [model]: type: decoder-only transformer, parameters: 175B, developers: OpenAI Entity registry contains 2 entities This context is injected before each new section to prevent inconsistency
By injecting the entity registry into each section's generation prompt, the model can avoid introducing contradictory attributes for entities it described earlier. If GPT-3 was described as having 175 billion parameters in section 1, the registry ensures that all subsequent sections see that fact before generating any text that might reference GPT-3's size.
The registry also handles aliases, which is important for natural writing style. Good writing avoids repeating the same noun phrase in every sentence. The model might refer to "BERT," "the encoder model," or "the model" in different sentences while meaning the same thing. The registry tracks these aliases so that when a later section uses "the encoder model," the generation system knows this is BERT and applies the correct attributes.
Consistency Checking
Even with proactive measures, long-form generation can produce inconsistencies. A post-hoc consistency check extracts factual claims from the generated text and verifies them against each other. This is a quality gate before the document is finalized.
import re
def check_numerical_consistency(text: str) -> list:
"""
Extract numerical claims from text and flag potential inconsistencies.
Returns list of (claim, numbers) tuples that need review.
"""
# Patterns to find sentences with numerical claims
patterns = [
r"\b(\d+(?:\.\d+)?)\s*(?:percent|%|billion|million|parameters)\b",
r"\bthere (?:are|were|is|was)\s+(\d+)\s+\w+",
r"\b(\d+(?:\.\d+)?)\s*(?:times|x)\s+(?:faster|slower|better|worse)",
]
claims = []
sentences = text.split(". ")
for sent in sentences:
for pattern in patterns:
matches = re.findall(pattern, sent, re.IGNORECASE)
if matches:
claims.append((sent.strip(), matches))
return claimsNumerical claims extracted: Numbers ['175'] in: 'GPT-3 has 175 billion parameters and was released in 2020' Numbers ['3'] in: 'There are 3 main approaches to fine-tuning' Numbers ['170'] in: 'GPT-3's 170 billion parameters make it one of the largest models' Numbers ['4'] in: 'There are 4 main approaches to fine-tuning.' Detected issues: - GPT-3's parameter count changed from 175B to 170B - 'Main approaches' count changed from 3 to 4 A consistency checker flags these sentences for human review.
More sophisticated consistency checking uses a language model to extract all factual claims from the document, then checks each claim against every other claim to detect contradictions. This can be implemented as a separate verification pass after the full document is generated. The checker generates pairs of claims and asks the model: "do these two statements contradict each other?" This approach catches semantic contradictions that simple pattern matching cannot detect, such as claiming a model was "released in early 2020" in one section and "made available in late 2019" in another.
Transition Generation
Coherence also requires smooth transitions between sections. When generating section-by-section, the boundaries between sections can feel abrupt. A reader working through the document experiences a jarring jump when one section ends and another begins without any bridging language. This is partly an artifact of the generation process itself: each section is generated independently, and the final sentence of section N is optimized without knowledge of what section N+1 will say.
A useful technique is to generate explicit transition sentences that bridge each pair of adjacent sections:
def generate_transition(
previous_section_last_paragraph: str,
next_section_title: str,
next_section_purpose: str,
) -> str:
"""Generate a bridging transition between two sections."""
prompt = f"""You are editing a long-form document. The previous section just ended with:
"{previous_section_last_paragraph}"
The next section is titled: "{next_section_title}"
Its purpose is: {next_section_purpose}
Write a single transition sentence (1-2 sentences) that smoothly connects these two sections.
The transition should make the logical connection explicit and give readers a reason to continue."""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0.6,
max_tokens=100,
)
return response.choices[0].message.contentTransition generation example: Previous section ends: '...the outline provides the skeleton that makes long-form generation tractable.' Next section: 'Hierarchical Generation' Generated transition: 'Building on this idea of structural planning, we can extend the outline approach to multiple levels of abstraction, creating a hierarchy that guides generation from the document level all the way down to the paragraph level.'
Transition generation is one of the highest-value post-processing steps for long-form text. A single sentence that explicitly connects two sections can transform a document that feels like a collection of disconnected essays into one that reads like a coherent argument building toward a conclusion.