Long-Form Generation › The Challenge of Length

Before diving into solutions, it helps to understand precisely why length creates difficulty. Three core problems arise as documents grow longer: context window exhaustion, topic drift, and factual inconsistency. Each of these is distinctive and requires different countermeasures.

Transformer models operate over fixed-length context windows. GPT-3 used 2,048 tokens; GPT-4 extended to 128,000 tokens; but even the largest windows eventually fill up. When a document exceeds the context window, the model loses access to its own earlier output. It cannot know what it wrote in section 2 when it is generating section 7. The result is repetition (restating points already made), contradiction (reversing claims from earlier), and incoherence (losing the thread of an argument).

Even within the context window, attention is not uniform. Research on long-context transformers has shown a "lost in the middle" effect: models tend to recall information near the beginning and end of context more reliably than information in the middle. Liu et al. (2023) demonstrated this empirically by inserting key facts at different positions within long contexts and measuring retrieval accuracy. Facts placed in the middle of the context were retrieved significantly less reliably than facts at the start or end, even when the full context was technically within the model's window. For long documents, critical structural information (the outline, the main thesis, earlier examples) can fall into this middle zone and become effectively invisible to later generation.

This has a practical implication: the most important structural information should be positioned at the start of each generation prompt, not buried in the middle of a long context dump. Outlines, entity registries, and high-level plans should come first, before the content they are meant to guide.

Without explicit structural anchors, language models drift. They tend to follow the most locally plausible continuation, which does not necessarily align with the globally intended direction of the document. A model generating an essay on climate change might gradually shift to discussing energy policy, then to geopolitics, then to economic theory, without any individual transition seeming wrong. Each sentence follows naturally from the last, but the document as a whole has abandoned its original focus.

This is partly a property of how language models are trained. The objective of next-token prediction rewards local coherence. The model learns to make the next sentence sound natural, not to maintain a document-level theme. The training signal for "this paragraph is on topic with respect to a document that started three thousand tokens ago" is weak or absent in standard language modeling. Fine-tuning on human-written long-form text helps, but does not fully solve the problem, because the model is still optimizing locally at generation time.

Topic drift is particularly insidious because it is invisible unless you read the document as a whole. Individual sections can each be well-written and internally coherent while the document collectively wanders far from where it started. Automated detection of drift requires comparing each section against a stable representation of the document's intended scope, which is exactly what outlines and plans provide.

In long documents, models frequently contradict themselves. A character's age changes between chapters. A technical claim stated in one section is reversed in another. A model generates "there are three main approaches" and then lists four. These consistency failures are harder to catch in long text because the contradicting sentences may be thousands of tokens apart.

Factual inconsistency is distinct from factual inaccuracy. A model can hallucinate consistently (always wrong about the same facts) or hallucinate inconsistently (giving different wrong answers in different sections). Long-form generation makes inconsistent hallucination more likely, because the model generates section content somewhat independently. Without a mechanism to track what was said earlier, later sections can freely contradict earlier ones.

The three challenges are interrelated. A model that runs out of context window cannot check its own earlier claims. A model that has drifted from its original topic may introduce facts relevant to the new topic that conflict with facts relevant to the original. And a model without a consistent structural plan may redraw the same entity with different properties each time it appears. Effective long-form generation must address all three problems together.