Decoding Temperature › Limitations and Impact

Temperature is powerful but imperfect. It is one of the oldest and most universally used generation controls, and it shapes the user experience of virtually every language model deployment. Understanding its limitations helps you know when to rely on it and when to supplement it with other techniques.

The most basic limitation of temperature is that it is a single scalar applied uniformly to all logits. This uniformity means temperature cannot make fine-grained distinctions within the vocabulary. Consider a medical question-answering system where you want diverse phrasing (how the answer is expressed) but not diverse facts (what claims are made). Temperature cannot distinguish between these: raising temperature increases variety in both phrasing and factual content simultaneously. A higher temperature that produces pleasantly varied sentence structures will equally raise the probability of factually incorrect medical claims. This creates a real tension in high-stakes domains where accuracy matters but repetitive outputs frustrate users.

This uniformity limitation also manifests in vocabulary-level problems. Some token types are sensitive to temperature in beneficial ways, like lexical variation in adjectives and adverbs, while others are sensitive in harmful ways, like proper nouns and factual entities. When you generate "The capital of France is," you want temperature to vary how the information is phrased (perhaps "Paris, the City of Light" vs. "Paris, the French capital"), but not to introduce uncertainty about which city is the capital. Temperature applies the same scaling to the logit for "Paris" and the logit for "Lyon," so any temperature above zero gives "Lyon" a non-zero chance of being selected. In practice, models usually have large enough logit gaps for factual entities that low-to-moderate temperatures do not introduce errors, but this provides no formal guarantee.

Temperature also interacts poorly with very long generation. Early tokens sampled at high temperature can push the model into unfamiliar territory, leading to compounding errors as generation proceeds. A single unusual word choice in token 5 might make token 50 completely incoherent, because the model's probability distributions are conditioned on everything that came before. The model was trained on coherent text and may produce high-quality continuations of any coherent prefix, but if temperature sampling introduces an incoherent prefix early on, the model has no mechanism to "recover" and get back on track. This is why many practitioners combine temperature with other techniques like top-k or nucleus sampling, covered in the following chapters, which constrain the damage from high-temperature sampling by preventing the model from assigning meaningful probability to the most implausible tokens.

Temperature can also interact unexpectedly with certain training techniques. Models trained with reinforcement learning from human feedback (RLHF) sometimes have sharpened distributions on "safe" responses and flattened distributions on "unsafe" responses. Raising temperature on such a model may disproportionately increase the probability of off-policy or harmful content, because the RLHF training suppressed those outputs without eliminating them, and temperature can partially resurrect them. Practitioners using RLHF-trained models often combine temperature with system-level safety constraints rather than relying on temperature alone to ensure output quality.

Another subtle limitation: temperature is applied at the individual token level, not at the sequence or sentence level. Even at $T = 0.7$, the model might produce a sentence that is locally coherent (each token follows from the previous) but globally incoherent (the overall sentence does not make sense). The local nature of autoregressive generation means that temperature controls randomness at each individual step without any mechanism for global coherence checking. Beam search and other sequence-level decoding strategies attempt to address this by considering multiple candidate sequences simultaneously, but temperature applies to single-step sampling.

Despite these limitations, temperature fundamentally shaped how we interact with language models. Before temperature scaling became standard, language model outputs felt robotic and predictable. Temperature gave users a dial to explore the space of possible outputs, making language models feel more creative and less deterministic. The concept transfers beyond language: temperature-like parameters appear in image generation (diffusion model guidance scales), music synthesis, and other generative AI systems. The intuition that "higher temperature means more randomness" has become part of the basic vocabulary of generative AI, understood by practitioners across all modalities.

The success of temperature scaling also revealed something important about language model training. The fact that simply rescaling logits produces coherent but varied outputs suggests that models learn meaningful probability distributions over vocabulary. The relative ordering of token probabilities carries semantic information: "sunny" really is more appropriate than "elephant" after "The weather today is," and temperature preserves this ordering while adjusting the degree of concentration. A model with poorly calibrated logits (where the numerical values do not reflect true relative likelihoods) would produce bizarre behavior when temperature is adjusted, suggesting that large language models learn to assign logit values that are informative about the appropriateness of continuations.