Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Decoding Temperature › How Temperature Scaling Works

Michael BrenndoerferUpdated July 27, 202518 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 202Decoding Temperature › How Temperature Scaling Works

Reading section 2 of 11 · How Temperature Scaling Works · 18 minSection 2 of 11 · 18 min

Section 2 of 11 · 18 min read

Decoding Temperature › How Temperature Scaling WorksOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand temperature, we need to start with what language models output. When you feed a prompt into a model, the final linear layer does not produce probabilities directly. Instead, it outputs a vector of logits: raw, unbounded scores for every token in the vocabulary. A logit of 5.0 does not mean "5% probability." It is just a score showing relative preference. The token with the highest logit is the model's top choice, but we need a way to convert these scores into actual probabilities we can sample from.

The distinction between logits and probabilities matters a great deal in practice. Because logits are unbounded, they can be negative, very large, or very small. A logit of 10.0 and a logit of -5.0 exist on the same scale, but the absolute values carry no direct probabilistic meaning. Only the differences between logits are semantically meaningful: a logit difference of 5.0 means the model prefers one token over another by a certain degree, but by how much in probability terms depends on all the other logits in the vocabulary. This is why the normalization step is needed.

The final linear layer of a transformer language model produces these logits through a simple matrix multiplication: the hidden state vector at the last token position is multiplied by a weight matrix of shape (hidden dimension, vocabulary size). This produces one scalar per token in the vocabulary, representing how strongly the model's learned representation activates each output class. Because the weight matrix is unconstrained, the logits can take any real value, and they carry no built-in probabilistic interpretation until we apply softmax.

Here the softmax function enters the picture. Softmax takes a vector of arbitrary real numbers and turns them into a valid probability distribution: all values become positive and sum to exactly 1. The key mechanism is exponentiation: taking the exponential of each logit makes all values positive (since $e^x > 0$ for any $x$ ), and then dividing by the sum of all exponentiated values normalizes them into probabilities. For a vocabulary of $V$ tokens with logits $z_1, z_2, \ldots, z_V$ , the standard softmax computes:

P(i) = \frac{\exp(z_i)}{\sum_{j=1}^{V} \exp(z_j)}

where:

$z_i$ : the logit for token $i$ , which can be any real number
$\exp(z_i)$ : the exponential function applied to the logit, so the result is positive regardless of the sign of $z_i$
$\sum_{j=1}^{V} \exp(z_j)$ : the sum of all exponentiated logits across the full vocabulary, serving as a normalizing constant that ensures the output probabilities sum to 1

Why does this formula make sense? Notice that exponentiation is a monotone increasing function: if $z_i > z_j$ , then $\exp(z_i) > \exp(z_j)$ . This means the ordering of preferences is preserved: the token with the highest logit still gets the highest probability. The exponential also amplifies differences: a logit that is 1.0 higher than another gets $e^1 \approx 2.72$ times more probability mass before normalization. This amplification is what makes softmax sensitive to logit differences, which is exactly the property that temperature will manipulate.

Out[4]:

Visualization

Bar chart showing logits ranging from 0.3 to 3.2 for ten weather-related tokens. — Raw logits from the model's output layer for ten weather-related tokens. The values are unbounded real numbers; higher logits indicate stronger preference but do not directly represent probabilities.

Bar chart showing corresponding probabilities from 0.02 to 0.32 after softmax. — Probabilities after softmax transformation. The exponential amplifies differences between logits, so the top token captures a disproportionately large share of probability mass.

The left panel shows raw logits: a descending sequence of numbers from 3.2 down to 0.3. These numbers are the direct output of the model's final linear layer and have no probabilistic interpretation on their own. The right panel shows what happens after softmax: the top token, "sunny," captures about 32% of the probability mass, while "unpredictable" at the bottom receives only about 2%. Notice that the logit difference between "sunny" (3.2) and "cloudy" (2.8) is just 0.4, yet softmax converts this into a meaningful gap in probability. The model prefers "sunny" by a factor of $e^{0.4} \approx 1.49$ relative to "cloudy," which becomes visible in the probability bars.

But here is the problem: the standard softmax gives us exactly one distribution. It reflects whatever probability concentrations the model learned during training. What if we want more control? What if the model's top choice is good but we want to explore alternatives? Or conversely, what if we want to make the model more decisive, committing more strongly to its best guess? The model's learned distribution is fixed at inference time, but we are free to transform it before sampling.

Introducing the Temperature ParameterLink Copied

Temperature gives us that control. The idea is elegantly simple: before applying softmax, we divide all logits by a temperature parameter $T$ . This single modification lets us reshape the entire probability distribution without changing the model's weights or the ordering of its preferences. All we are doing is rescaling the inputs to the softmax function, which changes how sharply it distinguishes between high-logit and low-logit tokens.

Temperature Scaling

Temperature scaling divides the logits by a temperature parameter $T$ before computing the softmax. Given logits $z_i$ for vocabulary token $i$ , the temperature-scaled probability is:

P(i) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

where:

$z_i$ : the logit (raw score) for token $i$ output by the model's final layer
$T$ : the temperature parameter, a positive scalar that controls distribution sharpness
$\exp(z_i / T)$ : the exponential of the scaled logit, which ensures positive values
$\sum_j \exp(z_j / T)$ : the sum over all tokens in the vocabulary, serving as a normalizing constant

When $T = 1$ , this reduces to the standard softmax. When $T < 1$ , dividing by a fraction amplifies the logit differences. When $T > 1$ , dividing by a larger number compresses the differences.

Why does this formula make sense? Notice that dividing $z_i$ by $T$ is equivalent to multiplying the entire logit vector by $1/T$ before softmax. When $T < 1$ , the factor $1/T > 1$ , so every logit gets scaled up. But because softmax only cares about differences between logits, the effect is to amplify those differences. When $T > 1$ , the factor $1/T < 1$ , so differences shrink. The normalization ensures the output still sums to 1 regardless of $T$ .

The name "temperature" comes from statistical mechanics, where a similar parameter controls the randomness of particle states in a physical system. At low temperature, particles settle into their lowest-energy states, creating a highly ordered, predictable configuration. At high temperature, particles have enough thermal energy to explore many states more freely, creating a more disordered, random configuration. The analogy carries over perfectly to language models: low temperature means the model commits to its top choices; high temperature means it explores more freely across the vocabulary. Think of temperature as the amount of "creative heat" you are injecting into the generation process.

Historical Context

Temperature scaling has its roots in the Boltzmann distribution from statistical mechanics, named after physicist Ludwig Boltzmann. In that context, the probability of a system occupying a state with energy $E$ at absolute temperature $T$ is proportional to $\exp(-E/k_BT)$ , where $k_B$ is the Boltzmann constant. At low temperatures, systems overwhelmingly occupy low-energy states; at high temperatures, higher-energy states become accessible.

The connection to language models was made explicit in the 1980s through work on Boltzmann machines, a class of stochastic neural networks where temperature controlled the probability of neuron activations. Geoffrey Hinton and Terry Sejnowski introduced the idea of annealing the temperature during training, starting high (random exploration) and cooling down (committing to good solutions). This technique, called simulated annealing, was borrowed from metallurgy, where controlled cooling of metals produces stronger crystal structures.

When softmax-based language models arrived in the 1990s and 2000s, the temperature parameter transferred naturally. The connection to the Boltzmann distribution is not merely metaphorical: the softmax function with a temperature parameter is mathematically identical to the Boltzmann distribution if you interpret the logits as negative energies. This deep connection between statistical mechanics and neural network training explains why "temperature" became the canonical term for this parameter.

A Concrete Example: Three Candidate TokensLink Copied

Let's make this concrete. Suppose a language model is completing the prompt "The capital of France is" and has narrowed down to three plausible tokens: "Paris," "Lyon," and "Marseille." The model outputs logits of 2.0, 1.0, and 0.5 respectively. Paris has the highest logit, so it is the model's top choice, but how do the probabilities change as we vary temperature?

In[5]:

Code

# Example logits for three tokens
logits = torch.tensor([2.0, 1.0, 0.5])
tokens = ["Paris", "Lyon", "Marseille"]


def temperature_softmax(
    logits: torch.Tensor, temperature: float
) -> torch.Tensor:
    """Apply temperature scaling and compute softmax."""
    scaled_logits = logits / temperature
    return F.softmax(scaled_logits, dim=0)

Out[6]:

Console

Token probabilities at different temperatures:

Token        T=0.5   T=1.0   T=2.0 
---------------------------------------------
Paris        0.844  0.629  0.481
Lyon         0.114  0.231  0.292
Marseille    0.042  0.140  0.227

The results reveal temperature's effect clearly. At $T = 0.5$ , "Paris" dominates with roughly 84% probability, leaving only scraps for the alternatives. The model is highly confident, almost deterministic. At $T = 1.0$ (standard softmax), we get the baseline distribution: Paris around 59%, Lyon around 22%, Marseille around 13%. The model still prefers Paris but gives meaningful weight to alternatives. At $T = 2.0$ , the distribution flattens dramatically: Paris drops toward 39%, and even Marseille climbs toward 22%. The model is now much more willing to sample less-preferred tokens.

The point is that the ordering of probabilities is preserved across all temperatures. Paris is always the most likely token, Lyon is always second, and Marseille is always third. Temperature does not change which tokens are preferred; it only changes how strongly the preferences are expressed. This is a important property: you are not distorting the model's knowledge when you adjust temperature, you are only adjusting how decisively it acts on that knowledge. This preservation of rank ordering is what makes temperature a principled intervention rather than arbitrary noise injection.

The Mathematics of Sharpening and FlatteningLink Copied

Why does this simple division produce such dramatic effects? The key insight comes from examining how temperature affects the probability ratio between any two tokens. Understanding this ratio reveals the core mechanism at work.

Consider two tokens with logits $z_1$ and $z_2$ , where $z_1 > z_2$ (token 1 is preferred). Under temperature scaling, their probabilities are computed separately but share the same normalization constant. We can compute the ratio $P(1)/P(2)$ to understand how much more likely token 1 is compared to token 2:

P(1) = \frac{\exp(z_1/T)}{\sum_j \exp(z_j/T)}, \quad P(2) = \frac{\exp(z_2/T)}{\sum_j \exp(z_j/T)}

When we compute the ratio $P(1)/P(2)$ , something beautiful happens: the normalization constants cancel out completely:

\frac{P(1)}{P(2)} = \frac{\exp(z_1/T)}{\exp(z_2/T)} = \exp\left(\frac{z_1 - z_2}{T}\right)

where:

$z_1 - z_2$ : the logit gap between the two tokens, representing how much more the model prefers token 1
$T$ : temperature, which scales how strongly this preference translates to a probability ratio
$\exp(\cdot)$ : the exponential function, which converts the scaled difference to a multiplicative ratio

This formula is the key to understanding temperature. The effective logit gap becomes $(z_1 - z_2)/T$ . Temperature acts as a divisor on the gap itself, not on the probabilities directly. The probability ratio between any two tokens is governed entirely by their logit difference and the temperature, independent of all other tokens in the vocabulary.

When $T < 1$ , we divide the gap by a fraction, which amplifies it. A logit difference of 1.0 at $T = 0.5$ becomes an effective difference of $1.0 / 0.5 = 2.0$ . The exponential of 2.0 is about 7.4, so the preferred token becomes 7.4 times more likely than its competitor. The distribution sharpens dramatically.

When $T > 1$ , we divide the gap by a number greater than 1, which compresses it. The same logit difference of 1.0 at $T = 2.0$ becomes an effective difference of $1.0 / 2.0 = 0.5$ . The exponential of 0.5 is about 1.65, so the preferred token is only 1.65 times more likely. The distribution flattens substantially. Notice that a token that was 2.72 times more likely at $T = 1.0$ (the baseline) compresses to just 1.65 times more likely at $T = 2.0$ : a significant change in behavior from a modest parameter adjustment.

An important consequence of this ratio formula is that temperature affects all token pairs simultaneously and by the same multiplicative factor. If the ratio between token A and token B is multiplied by $k$ when temperature changes, the ratio between token C and token D is also multiplied by $k$ (adjusted for the difference in their logit gaps). This uniformity is both a feature and a limitation: it produces simple, predictable behavior but cannot be tailored to treat different vocabulary regions differently.

Let's verify this with actual calculations:

In[7]:

Code

def probability_ratio(z1: float, z2: float, temperature: float) -> float:
    """Compute probability ratio between two tokens given their logits."""
    return np.exp((z1 - z2) / temperature)


# Logit gap of 1.0 between tokens
z1, z2 = 2.0, 1.0

Out[8]:

Console

Probability ratio (P(token1) / P(token2)) at different temperatures:

T = 0.25: ratio =    54.60
T = 0.5 : ratio =     7.39
T = 1.0 : ratio =     2.72
T = 2.0 : ratio =     1.65
T = 4.0 : ratio =     1.28

Out[9]:

Visualization

Curve showing probability ratio decreasing from over 50 at T=0.25 to near 1 at T=4, with horizontal dashed line at ratio=1. — Probability ratio between two tokens as temperature varies. For a fixed logit gap of 1.0, the ratio equals e at T=1.0 (the baseline). Low temperatures amplify the gap exponentially, making one token many times more likely; high temperatures compress the gap toward a ratio of 1 (equal probability).

The numbers tell the story. At $T = 0.25$ , the higher-probability token is 55 times more likely than its competitor, an overwhelming advantage. At $T = 1.0$ (baseline), the ratio is 2.72 (which is just $e^1$ , exactly as the formula predicts). At $T = 4.0$ , that ratio shrinks to just 1.28, meaning the two tokens are nearly equally likely despite the original logit gap. Temperature truly acts as a dial between certainty and randomness.

Temperature Extremes: The Limiting CasesLink Copied

To build complete intuition, consider what happens at the mathematical extremes. These limiting cases are never exactly achievable but reveal the asymptotic behavior of temperature scaling.

As $T \to 0$ (approaching zero):

The effective logit gap $(z_1 - z_2)/T$ grows without bound for any non-zero gap. The exponential of infinity is infinity, so the probability ratio between the top token and any other token becomes infinite. In practice, this means all probability mass concentrates on the single highest-logit token. This limiting case is equivalent to greedy decoding or argmax selection: always pick the most likely token, with zero randomness. Think of it as a model that has become infinitely confident in its first guess, never entertaining any alternative.

Greedy decoding has a specific failure mode worth understanding: repetition loops. If the model is in a context where a particular phrase is highly probable, greedy decoding will produce that phrase deterministically, and because the phrase is now part of the context, the same phrase becomes highly probable again. The result is the infamous "degenerate repetition" problem: "The cat sat on the mat. The cat sat on the mat. The cat sat on the mat." A small positive temperature (0.1 to 0.3) typically eliminates this problem while preserving most of the benefits of low-temperature generation.

As $T \to \infty$ (approaching infinity):

The effective logit gap $(z_1 - z_2)/T$ shrinks toward zero for any finite gap. The exponential of zero is 1, so the probability ratio between any two tokens approaches 1:1. All tokens become equally likely regardless of their original logits. The distribution approaches uniform, and generation becomes pure random sampling from the vocabulary. Think of it as a model that has forgotten everything it learned, selecting tokens as if by rolling a many-sided die.

Neither extreme is useful for text generation. $T \approx 0$ produces repetitive, predictable text that lacks nuance. The model keeps selecting the same high-probability continuations, often getting stuck in loops where the same phrases repeat endlessly. $T \approx \infty$ produces incoherent gibberish since token selection ignores the model's learned preferences entirely. The resulting text looks like random strings drawn from a vocabulary distribution, violating every grammatical and semantic constraint the model learned during training. Practical values fall between these extremes, typically in the range 0.1 to 2.0.

Out[10]:

Visualization

Bar chart showing Paris at 99.3% probability with Lyon and Marseille near zero. — At T=0.1, nearly all probability mass concentrates on Paris (greedy behavior). Lyon and Marseille are negligible, making generation essentially deterministic.

Bar chart showing Paris at 59%, Lyon at 22%, Marseille at 13%. — At T=1.0 (standard softmax), the model's learned preferences are preserved. Paris leads but alternatives receive meaningful probability mass.

Bar chart showing nearly equal probabilities around 33% for all three tokens. — At T=10.0, the distribution approaches uniformity. All three tokens receive nearly equal probability, making generation almost random.

The visualization makes the extremes clear. At $T = 0.1$ , Paris captures nearly all of the probability mass. This leaves virtually nothing for alternatives. The generation outcome is essentially predetermined before any randomness is applied. At $T = 10.0$ , the three tokens have nearly equal probability. It approaches the uniform distribution of 33.3% each. If you sampled many tokens at this temperature, you would sample each of the three cities with roughly equal frequency, which is obviously not the behavior you want for a factual geography question. The standard softmax at $T = 1.0$ sits in between. This reflects the model's learned preferences while still letting meaningful sampling diversity.