Decoding Temperature › Key Parameters

When implementing temperature-controlled sampling, these parameters determine behavior:

• temperature (float, typically 0.1-2.0): The scaling factor applied to logits before softmax. Values below 1.0 sharpen the distribution, making high-probability tokens more dominant. Values above 1.0 flatten the distribution, giving lower-probability tokens more chance. A value of 1.0 preserves the original learned distribution.

• do_sample (bool): Whether to sample from the distribution (True) or use greedy decoding (False). Temperature only affects output when do_sample=True. With do_sample=False, the model always selects the highest-probability token regardless of temperature.

• top_k (int, 0 to disable): When combined with temperature, restricts sampling to the top k most probable tokens. Setting top_k=0 disables this constraint, letting temperature to affect the full vocabulary distribution.

• top_p (float, 0.0-1.0): Nucleus sampling threshold, often used alongside temperature. Setting top_p=1.0 disables nucleus sampling, isolating the temperature effect. Lower values restrict sampling to tokens whose cumulative probability reaches the threshold.

• max_new_tokens (int): Maximum number of tokens to generate. Longer sequences at high temperature tend to accumulate errors, so consider lower temperatures for longer outputs.

For most applications, start with temperature=0.7 and adjust based on output quality. Decrease if outputs are too random or incoherent; increase if outputs are too repetitive or predictable.