Constrained Decoding › Constrained Decoding

Language models generate text by sampling from a probability distribution over the vocabulary at each step. This works well for creative writing, conversational responses, and open-ended generation. But what happens when you need the model to produce structured output, like valid JSON, SQL queries, or text matching a specific pattern? Standard sampling offers no guarantees. The model might produce syntactically invalid JSON, hallucinate field names, or ignore your carefully specified schema.

Constrained decoding solves this problem by modifying the generation process itself. Instead of allowing the model to choose freely from all tokens at each step, we restrict it to tokens that keep the output on a valid path. The model still uses its learned distributions, but we guide it toward outputs that satisfy our structural requirements. This technique has become essential for building reliable AI systems that integrate with existing software infrastructure.