Decoding Temperature › Decoding Temperature

When a language model predicts the next token, it does not simply look up the correct answer in a table. It produces a full probability distribution over its entire vocabulary, every single word and subword it has ever encountered during training. Given the prompt "The capital of France is," the model might assign 0.85 to "Paris," 0.05 to "Lyon," 0.02 to "Marseille," and tiny probabilities to thousands of other tokens. That distribution encodes everything the model learned during training: the frequencies of words, the syntactic patterns of sentences, the semantic constraints of meaning. Temperature is the parameter that controls how we interpret and sample from this distribution at generation time.

Temperature answers a basic question: how much should we trust the model's probability rankings? A temperature of 1.0 preserves the learned distribution exactly, sampling from it just as the model's training implied. Lower temperatures sharpen the distribution, making high-probability tokens even more likely and pushing the model toward deterministic, predictable output. Higher temperatures flatten the distribution, giving lower-probability tokens a fighting chance and introducing creative variability. This single scalar parameter is, remarkably, one of the most important controls practitioners have over language model behavior.

Think of temperature as a confidence dial. At one extreme, you tell the model to always go with its gut, committing to whichever continuation it found most plausible. At the other extreme, you tell the model to treat all of its guesses as equally valid and pick one at random. In between lies a continuous spectrum of behaviors: cautious and factual at the low end, expressive and exploratory at the high end. No other single parameter gives you quite this direct a handle on the randomness of generation. Unlike architectural choices or fine-tuning procedures, temperature is a real-time knob you can adjust on every inference call.

Understanding temperature matters for both practitioners who deploy language models and researchers who study them. If you are building a factual question-answering system, temperature near zero ensures you get the most probable (and often most accurate) answer without unnecessary variability. If you are building a creative writing assistant, moderate temperature is what separates outputs that feel alive from outputs that feel like template completions. And if you are studying how language models generalize, temperature provides a window into the shape of the probability distributions they have learned. A model with well-calibrated logits will produce coherent outputs across a wide range of temperatures; a poorly calibrated model may degrade rapidly.

This chapter explores temperature from first principles. You will learn the mathematical mechanics of temperature scaling, visualize how it reshapes probability distributions, implement temperature-controlled sampling, and develop intuition for selecting appropriate values across different generation tasks. By the end, you will understand how to set the temperature knob, why changing it has the effects it does, and when those effects are beneficial versus harmful.