Long-Form Generation › Evaluation of Long-Form Text
Language AI HandbookAll chapters Long-Form Generation › Evaluation of Long-Form TextOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Evaluating long-form generation is substantially harder than evaluating short-form generation. Standard metrics like BLEU and ROUGE, designed for sentence or paragraph comparison, break down at document scale. We need evaluation methods that capture structure, coherence, and factual accuracy across the full document.
The evaluation challenge reflects a deeper problem: long-form text quality is multidimensional. A document can be fluent but incoherent, coherent but incomplete, complete but factually wrong. No single metric captures all these dimensions, and the dimensions can be partially independent. A system that maximizes ROUGE scores might sacrifice coherence; a system optimized for coherence might produce coherent-but-wrong content. A thorough evaluation therefore requires multiple complementary metrics.
Reference-Based Metrics at Document Scale
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram overlap between generated and reference text. For long-form generation, several problems arise.
First, reference texts are rare. Unlike translation datasets where many correct translations exist, long-form generation tasks often have only one reference document per prompt, and that reference reflects just one of many valid ways to structure the content. Second, ROUGE does not measure structure. A document with the right words in the wrong order can score well on ROUGE while being incoherent. Third, ROUGE does not scale linearly with document quality. A document that nails the first three sections and completely fails the last three might score identically to one that mediocrely covers all six sections.
The solution is to compute ROUGE at the section level rather than the document level. Section-level ROUGE reveals which parts of a document succeeded and which failed, giving diagnostic value that document-level aggregates cannot.
# uv pip install rouge-score
try:
from rouge_score import rouge_scorer
except ImportError:
from collections import Counter
from types import SimpleNamespace
def _ngram_f1(reference, candidate, n):
ref_tokens = reference.lower().split()
cand_tokens = candidate.lower().split()
ref = Counter(
tuple(ref_tokens[i : i + n]) for i in range(len(ref_tokens) - n + 1)
)
cand = Counter(
tuple(cand_tokens[i : i + n])
for i in range(len(cand_tokens) - n + 1)
)
overlap = sum((ref & cand).values())
precision = overlap / max(sum(cand.values()), 1)
recall = overlap / max(sum(ref.values()), 1)
return 2 * precision * recall / max(precision + recall, 1e-12)
def _lcs_f1(reference, candidate):
ref = reference.lower().split()
cand = candidate.lower().split()
row = [0] * (len(cand) + 1)
for ref_token in ref:
previous = row[:]
for j, cand_token in enumerate(cand, start=1):
row[j] = (
previous[j - 1] + 1
if ref_token == cand_token
else max(previous[j], row[j - 1])
)
precision = row[-1] / max(len(cand), 1)
recall = row[-1] / max(len(ref), 1)
return 2 * precision * recall / max(precision + recall, 1e-12)
class _FallbackRougeScorer:
def __init__(self, metrics, use_stemmer=True):
self.metrics = metrics
def score(self, reference, candidate):
values = {
"rouge1": _ngram_f1(reference, candidate, 1),
"rouge2": _ngram_f1(reference, candidate, 2),
"rougeL": _lcs_f1(reference, candidate),
}
return {
metric: SimpleNamespace(fmeasure=values[metric])
for metric in self.metrics
}
rouge_scorer = SimpleNamespace(RougeScorer=_FallbackRougeScorer)
def evaluate_long_form_rouge(
generated_sections: list, reference_sections: list
) -> dict:
"""
Compute ROUGE scores at the section level for long-form documents.
Returns per-section and aggregate scores.
"""
scorer = rouge_scorer.RougeScorer(
["rouge1", "rouge2", "rougeL"], use_stemmer=True
)
section_scores = []
for gen, ref in zip(generated_sections, reference_sections):
scores = scorer.score(ref, gen)
section_scores.append(
{
metric: scores[metric].fmeasure
for metric in ["rouge1", "rouge2", "rougeL"]
}
)
# Aggregate across sections
aggregate = {}
for metric in ["rouge1", "rouge2", "rougeL"]:
values = [s[metric] for s in section_scores]
aggregate[metric] = {
"mean": sum(values) / len(values),
"min": min(values),
"max": max(values),
}
return {"per_section": section_scores, "aggregate": aggregate}Simulated ROUGE scores for a 4-section document: Section ROUGE-1 ROUGE-2 ---------------------------------- Section 1 0.680 0.410 Section 2 0.710 0.380 Section 3 0.450 0.220 Section 4 0.620 0.350 Mean ROUGE-1: 0.615 Mean ROUGE-2: 0.340 Section 3 has notably lower scores (0.45 / 0.22), suggesting it diverged most from the reference content.
Section-level scoring immediately highlights problems that would be hidden in a document-level average. In this example, Section 3's ROUGE-2 score of 0.22 is roughly half the score of Section 2 (0.38). A document-level average of 0.34 would obscure this gap and suggest uniformly mediocre performance across all sections.
Coherence Scoring
Coherence scoring measures how well a document holds together as a unified text, independent of any reference document. This is particularly valuable when reference texts are unavailable, as is common in practice.
The entity grid model, developed by Barzilay and Lapata (2008), represents each document as a grid where rows are sentences and columns are entities, with cells indicating whether an entity appears as the grammatical subject, object, or not at all in each sentence. Coherent documents tend to have entities appearing in consistent grammatical roles across adjacent sentences, while incoherent documents show erratic patterns. A coherent entity chain might look like: subject, subject, object, not-present, not-present, subject. An incoherent chain might look like: subject, not-present, object, subject, not-present, object with no discernible pattern.
More recently, neural coherence models fine-tune language models on the task of distinguishing coherent documents from artificially scrambled versions. A model trained on this task can assign coherence scores to generated documents without requiring reference texts. The training data is cheap to produce: any existing document provides both a coherent version (the original) and an incoherent version (sentences in random order).
def compute_entity_transition_score(text: str) -> float:
"""
Simplified entity transition coherence scoring.
Measures the proportion of sentence pairs that share at least one entity,
as a proxy for local coherence.
"""
sentences = [s.strip() for s in text.split(".") if s.strip()]
def extract_entities(sentence: str) -> set:
# Simplified: extract capitalized words as proxy for named entities
words = sentence.split()
return {w.lower() for w in words if w[0].isupper() and len(w) > 2}
if len(sentences) < 2:
return 0.0
shared_entity_pairs = 0
for i in range(len(sentences) - 1):
entities_a = extract_entities(sentences[i])
entities_b = extract_entities(sentences[i + 1])
if entities_a & entities_b: # Non-empty intersection
shared_entity_pairs += 1
return shared_entity_pairs / (len(sentences) - 1)Entity Transition Coherence Scores: Coherent text: 0.667 Incoherent text: 0.000 Higher score = more adjacent sentence pairs share entities Coherent text shows consistent entity chains across sentences; incoherent text introduces unrelated entities at each step.
The simplified entity transition score separates the two texts clearly. In the coherent text, adjacent sentences share entity references (transformer, attention, NLP), signaling topical continuity. In the incoherent text, each sentence introduces entirely new entities with no connection to the previous sentence.
Coverage and Completeness
For documents generated from a given topic or prompt, we also want to measure coverage: does the document address what it was supposed to address? Coverage metrics compare the set of topics or subtopics in the generated document against a predefined set of required topics.
Coverage evaluation is particularly useful in applied settings where the required content is known in advance, such as generating a report on a fixed set of questions or a tutorial covering a specified list of concepts. The metric is less useful for creative or open-ended generation, where the required content is not predefined.
def compute_topic_coverage(
generated_text: str, required_topics: list, threshold: float = 0.6
) -> dict:
"""
Measure what fraction of required topics are covered in the generated text.
Uses keyword matching as a proxy for topic coverage.
"""
generated_lower = generated_text.lower()
topic_coverage = {}
for topic in required_topics:
# Topic is "covered" if majority of its words appear in the text
topic_words = topic.lower().split()
words_found = sum(1 for w in topic_words if w in generated_lower)
coverage_ratio = words_found / len(topic_words)
topic_coverage[topic] = coverage_ratio >= threshold
covered_count = sum(topic_coverage.values())
total = len(required_topics)
return {
"topics": topic_coverage,
"coverage_ratio": covered_count / total,
"covered": covered_count,
"total": total,
"missing": [t for t, v in topic_coverage.items() if not v],
}Topic Coverage Analysis:
Overall coverage: 75.0% (6/8 topics)
Per-topic coverage:
COVERED outline generation
COVERED hierarchical decomposition
COVERED entity tracking
COVERED coherence scoring
COVERED ROUGE evaluation
COVERED context window limitations
MISSING transition generation
MISSING factual consistency
Topics needing more coverage: transition generation, factual consistencyCoverage analysis lets writers and systems identify gaps before finalizing a document. In the example above, "transition generation" and "factual consistency" are flagged as uncovered, prompting the generator to either add sections on those topics or revise existing sections to address them more explicitly.
Human Evaluation Protocols
Automatic metrics, while useful, cannot fully capture document quality. Human evaluation remains the gold standard for long-form generation. Standard human evaluation protocols for long-form text ask evaluators to rate documents on several dimensions.
The key dimensions typically assessed include:
- Coherence: Does the document flow logically from beginning to end? Do sections connect naturally?
- Relevance: Does the document stay on topic and address what was asked?
- Factual accuracy: Are the claims in the document true and self-consistent?
- Fluency: Is the writing grammatically correct and readable at the sentence level?
- Completeness: Does the document fully address the topic, or are important aspects missing?
Human evaluation is expensive, slow, and subject to annotator disagreement, but it remains the most reliable way to assess whether a long-form document serves its intended purpose. Inter-annotator agreement for long-form text quality is typically lower than for shorter tasks, partly because different evaluators weight the dimensions differently. A researcher might prioritize factual accuracy; a general reader might prioritize fluency; an editor might prioritize structural coherence. This variability makes standardized human evaluation protocols important for ensuring comparability across studies.
Recent work has explored using large language models (LLMs) as evaluators, asking them to rate generated text on the same dimensions that human annotators assess. LLM-based evaluation can be faster and cheaper than human evaluation while showing moderate correlation with human judgments, but it inherits the models' biases, including a well-documented tendency to prefer longer and more verbose text regardless of actual quality.