Text Generation Applications › Generation Quality Dimensions

Quality in text generation is multi-dimensional. No single score captures whether output is good, and different applications weight dimensions differently. Understanding these dimensions allows you to design better prompts and evaluation pipelines.

Fluency measures how natural and grammatically correct generated text reads. Modern large language models almost always produce fluent text; fluency failures are rare except at very low temperatures that cause repetition or very high temperatures that cause incoherence. Fluency is necessary but not sufficient.

It is worth understanding why fluency is so reliable in modern models. The causal language modeling objective, applied over billions of tokens of well-written text, powerfully incentivizes grammatical correctness: an ungrammatical continuation is almost always lower probability than a grammatical one. Models that cannot produce grammatical text have high perplexity on standard benchmarks, which pushes against them during training. Fluency is, in a sense, the dimension that training data quality most directly addresses.

Relevance measures how well the output addresses the prompt. A fluent but irrelevant response fails the task. Relevance depends heavily on prompt quality: vague prompts produce tangentially relevant outputs; precise prompts with explicit constraints produce tighter results.

A common relevance failure is scope creep: the model addresses the prompt but adds substantial content that is adjacent to but outside the requested scope. A prompt asking for a 100-word explanation of gradient descent may produce a 300-word explanation that covers gradient descent, its variants, applications to neural networks, and comparison to second-order methods. Each individual claim may be accurate, but the output does not comply with the task. Explicit length constraints and instructions to focus exclusively on the stated topic reduce scope creep.

Another relevance failure is question reframing: the model subtly rewrites the question it was asked before answering, producing a response that is internally coherent but does not address the original prompt. This happens because the model generates both its interpretation of the task and its response to that interpretation sequentially. Prompts that ask the model to state its interpretation before answering, or that provide explicit markers for the input task, help catch this failure early.

Factual accuracy is the dimension where generation most commonly fails. Models generate based on distributional patterns in training data, not by looking up ground truth. They can produce convincing, fluent, completely false statements. This is the hallucination problem covered in detail in the Hallucination chapters.

Hallucination is not uniformly distributed. Models are more likely to hallucinate when asked about specific facts (dates, statistics, citations, names of obscure entities) than when asked about well-established general knowledge. They are more likely to hallucinate when the requested information is rare in the training set than when it is common. They are more likely to hallucinate in domains (medicine, law, finance) where training data may contain contradictory or outdated information. Understanding these patterns allows practitioners to identify which parts of a generation pipeline require retrieval augmentation or human verification.

For content creation applications where factual accuracy matters, the standard mitigation is retrieval-augmented generation (RAG). The model is given relevant retrieved documents as context and instructed to base its output on them. We covered RAG in depth in the RAG chapters. RAG does not eliminate hallucination, but it substantially reduces it for facts that appear in the retrieved documents. The model can still hallucinate by misquoting the retrieved context, conflating information from different documents, or generating claims beyond the scope of the retrieved context.

Coherence measures whether the text holds together as a unified piece: whether it maintains consistent facts, does not contradict itself, and follows a logical structure. Long outputs challenge coherence more than short ones. Hierarchical generation strategies and explicit document outlines improve coherence.

Coherence also has a subtle intra-session dimension. When a model generates a multi-section document in a single call, it may drift in its terminology: the first section calls something "token embedding" and the third section calls the same concept "input representation." These are not factual errors, but they reduce readability and suggest the text was generated without a unified plan. Instructing the model to maintain a defined glossary, or post-processing outputs to normalize terminology, addresses this failure mode.

Logical coherence is a more demanding criterion than terminological coherence. An argument that makes a claim in one paragraph, then makes a supporting claim in the next paragraph that contradicts the first, is logically incoherent even if every sentence is individually grammatical. Logical coherence checking requires semantic understanding of the content, which is difficult to automate. Models asked to critique their own outputs for logical consistency can catch some of these issues, but self-critique is unreliable for the same reasons that self-verification is unreliable: the model that generated the error often also fails to detect it.

Many applications impose hard constraints: word count limits, required inclusions or exclusions, stylistic rules, format requirements (JSON, markdown headers, bullet lists). Adherence measures whether the model respects these constraints. Models often violate constraints at the margins, particularly count constraints ("write exactly 100 words"). Verifying adherence programmatically and retrying on failure is standard practice.

Hard constraints form a hierarchy by verifiability. Word count is trivially verifiable by a tokenizer. Required phrase inclusion is verifiable by substring search. JSON format validity is verifiable by a JSON parser. Logical consistency and tone appropriateness are much harder to verify programmatically. Building a production constraint-checking pipeline means distinguishing between hard constraints that can be checked algorithmically and soft constraints that require semantic evaluation.

Diversity is a quality dimension in some applications. When generating multiple variants for A/B testing, you want outputs that are materially different, not paraphrases of the same content. Higher temperature increases diversity at the cost of reliability. Techniques like sampling with different random seeds, modifying system prompts, or using diverse persona descriptions improve output variety.

It is worth understanding why models tend to produce similar outputs when temperature is low. A low-temperature model concentrates probability mass on the few highest-probability tokens at each step. Since the same tokens tend to be highest probability across similar prompts, the model effectively follows the most common pattern in its training data, which produces similar outputs for similar prompts. Raising temperature introduces randomness that pulls the model off the most-trodden path, exploring less common but still plausible continuations. The trade-off is real: those less common paths are less common partly because they are less appropriate, not only because they are less frequent. This is why high-diversity applications require human filtering to discard the outputs that are different but worse.

A practical diversity enhancement technique is persona prompting: prefacing each generation with a different persona instruction ("You are a concise technical writer" vs. "You are an enthusiastic marketing copywriter") to push the model toward different stylistic registers. This reliably produces outputs that differ in tone and vocabulary even at moderate temperature, without the coherence risk of very high temperature.