Decoding Temperature › Summary

Temperature controls the sharpness of the probability distribution during language model sampling. By dividing logits by a temperature parameter $T$ before softmax, we can make the distribution more peaked (low $T$) or more uniform (high $T$). The mechanism is mathematically clean: temperature amplifies or compresses logit differences, which exponentiates into multiplicative changes in probability ratios.

Key takeaways:

• $T = 1.0$ preserves the learned distribution. Lower values sharpen it; higher values flatten it.
• $T \to 0$ approaches greedy decoding (argmax). $T \to \infty$ approaches uniform random sampling.
• Practical ranges typically fall between 0.1 and 2.0, with most applications using 0.3 to 1.2.
• Task matters: Factual and code generation prefer low temperature. Creative writing benefits from higher values.
• Temperature controls the quality-diversity trade-off: more diversity comes at the cost of coherence.
• Temperature affects all tokens uniformly, which can be limiting when you want selective diversity.
• The probability ratio between any two tokens is $\exp((z_1 - z_2)/T)$, so temperature acts by rescaling the effective logit gap between all pairs of tokens.
• Entropy of the output distribution increases monotonically with temperature. This provides a precise measure of how much randomness temperature introduces.

Temperature is often combined with top-k and nucleus (top-p) sampling to get finer control over the output distribution. These techniques, covered in the following chapters, truncate the distribution before sampling, preventing temperature from giving meaningful probability to highly unlikely tokens. The combination of temperature (which controls how peaked the distribution is) and top-p sampling (which cuts off the tail of the distribution) is one of the most effective and widely used generation strategies in practice.