Constrained Decoding › Summary

Constrained decoding turns language model generation from probabilistic text production into structured output generation. This chapter covers:

• Token masking forms the foundation: at each step, we zero out probabilities for tokens that would violate constraints, then sample from the remaining valid tokens
• Grammar-guided generation uses formal grammars (typically context-free grammars compiled to finite state machines) to specify valid output structures and compute valid tokens efficiently
• JSON schema constraints convert schemas into grammars that enforce both valid JSON syntax and adherence to the specified schema structure
• Regex constraints provide a lighter-weight alternative for pattern matching, converted to DFAs for efficient constraint checking
• Constrained beam search extends beam search to maintain multiple valid hypotheses, enabling recovery from locally suboptimal choices
• Combining with sampling strategies allows constraints to work alongside temperature, top-k, and nucleus sampling for controlled randomness within valid outputs

The technique has become essential for building reliable AI systems that must produce machine-readable output. When you need a language model to return valid JSON, generate syntactically correct code, or produce output matching any formal specification, constrained decoding provides the guarantee that pure prompting cannot.