Constrained Decoding › Key Parameters
Language AI HandbookAll chapters Constrained Decoding › Key ParametersOne section · Sections in this chapter11 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When implementing constrained decoding, several parameters control the behavior of the system:
-
temperature (float, typically 0.1-2.0): Controls randomness within valid tokens. Lower values make generation more deterministic, always choosing the highest-probability valid token. Higher values spread probability more evenly across valid options. Temperature of 1.0 preserves the model's original distribution.
-
top_k (int, typically 5-50): After filtering to valid tokens, further restricts sampling to the k highest-probability options. Reduces variance in generation by excluding low-probability valid tokens.
-
beam_width (int, typically 3-10): For constrained beam search, determines how many candidate sequences to maintain. Larger widths explore more paths but increase computation linearly.
-
max_length (int): Maximum number of tokens to generate. Prevents infinite loops when constraints allow arbitrarily long valid sequences.
-
constraint_type: The type of constraint to apply. Options include grammar-based (context-free grammars), schema-based (JSON schemas), regex-based (regular expressions), or custom constraint functions.
For production use with JSON output, libraries like Outlines handle these parameters automatically. For custom constraints, start with temperature=1.0 and no top_k filtering to preserve the model's preferences, then tune based on output quality.