Constrained Decoding › Why Unconstrained Generation Fails
Language AI HandbookAll chapters Constrained Decoding › Why Unconstrained Generation FailsOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into solutions, let's understand why asking a model nicely to follow a format often doesn't work. Language models learn patterns from training data, including the patterns of JSON and XML. They can often produce valid structured output. But "often" isn't good enough for production systems.
Consider a simple scenario: you want the model to output a JSON object with specific fields. Even with careful prompting, several failure modes emerge:
- Syntax errors: Missing commas, unmatched brackets, trailing commas that violate strict JSON
- Schema violations: Extra fields the model invented, missing required fields, wrong data types
- Encoding issues: Unescaped special characters, invalid Unicode sequences
- Format drift: The model starts with JSON but drifts into natural language explanation
import json
# Examples of common generation failures when asking for structured output
failed_outputs = [
# Missing closing brace
'{"name": "Alice", "age": 30',
# Extra field not in schema
'{"name": "Bob", "age": 25, "favorite_color": "blue"}',
# Wrong type (string instead of number)
'{"name": "Carol", "age": "twenty-five"}',
# Trailing comma (invalid JSON)
'{"name": "Dave", "age": 35,}',
# Mixed format - starts JSON, ends with explanation
'{"name": "Eve"} Note: I only included the name field since...',
]
def validate_json(text: str) -> tuple[bool, str]:
"""Check if text is valid JSON and return status with message."""
try:
json.loads(text)
return True, "Valid JSON"
except json.JSONDecodeError as e:
return False, f"Invalid: {e.msg}"Validation results for common model outputs:
✗ Invalid: Expecting ',' delimiter
Output: {"name": "Alice", "age": 30
✓ Valid JSON
Output: {"name": "Bob", "age": 25, "favorite_color": "blue...
✓ Valid JSON
Output: {"name": "Carol", "age": "twenty-five"}
✗ Invalid: Expecting property name enclosed in double quotes
Output: {"name": "Dave", "age": 35,}
✗ Invalid: Extra data
Output: {"name": "Eve"} Note: I only included the name fie...These failures aren't bugs in the model. They reflect the fundamental nature of autoregressive generation: each token is sampled independently given the context, with no mechanism to enforce global constraints. The model doesn't "know" it needs to close a brace three tokens from now. Constrained decoding adds that knowledge.
