Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Decoding Temperature › Worked Example: Step-by-Step Temperature Calculation

Michael BrenndoerferUpdated July 27, 20254 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 202Decoding Temperature › Worked Example: Step-by-Step Temperature Calculation

Reading section 4 of 11 · Worked Example: Step-by-Step Temperature Calculation · 4 minSection 4 of 11 · 4 min

Section 4 of 11 · 4 min read

Decoding Temperature › Worked Example: Step-by-Step Temperature CalculationOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before implementing temperature in a full generation loop, let's walk through a complete numerical example from raw logits to sampled token. This trace is the kind of calculation that happens millions of times during a single generation call: once per token position, across every possible token in the vocabulary.

Suppose our model has just processed the prompt "The weather in Paris is" and the final linear layer has output the following logits for five candidate tokens:

"beautiful": 3.5
"rainy": 2.0
"cold": 1.8
"unusual": 0.9
"enormous": -0.5

We will apply temperature $T = 0.7$ and trace every step.

Step 1: Divide all logits by the temperature.

\begin{aligned} z_{\text{beautiful}} / T &= 3.5 / 0.7 = 5.0 \\ z_{\text{rainy}} / T &= 2.0 / 0.7 \approx 2.857 \\ z_{\text{cold}} / T &= 1.8 / 0.7 \approx 2.571 \\ z_{\text{unusual}} / T &= 0.9 / 0.7 \approx 1.286 \\ z_{\text{enormous}} / T &= -0.5 / 0.7 \approx -0.714 \end{aligned}

Notice that the logit differences have all been amplified by a factor of $1/T = 1/0.7 \approx 1.43$ . The gap between "beautiful" and "rainy" was $3.5 - 2.0 = 1.5$ in the original logits, and now it is $5.0 - 2.857 = 2.143$ in the scaled logits.

Step 2: Exponentiate the scaled logits.

\begin{aligned} \exp(5.0) &\approx 148.41 \\ \exp(2.857) &\approx 17.40 \\ \exp(2.571) &\approx 13.08 \\ \exp(1.286) &\approx 3.62 \\ \exp(-0.714) &\approx 0.49 \end{aligned}

Step 3: Compute the normalization constant (denominator).

Z = 148.41 + 17.40 + 13.08 + 3.62 + 0.49 = 183.00

Step 4: Divide each exponentiated value by $Z$ to get probabilities.

\begin{aligned} P(\text{beautiful}) &= 148.41 / 183.00 \approx 0.811 \\ P(\text{rainy}) &= 17.40 / 183.00 \approx 0.095 \\ P(\text{cold}) &= 13.08 / 183.00 \approx 0.072 \\ P(\text{unusual}) &= 3.62 / 183.00 \approx 0.020 \\ P(\text{enormous}) &= 0.49 / 183.00 \approx 0.003 \end{aligned}

Step 5: Verify that probabilities sum to 1.

0.811 + 0.095 + 0.072 + 0.020 + 0.003 = 1.001 \approx 1.0

The small rounding error is expected; exact computation would give exactly 1.0.

Step 6: Sample from the distribution.

We now draw one token index according to these probabilities. With $T = 0.7$ , "beautiful" has an 81.1% chance of being selected. If we had used $T = 1.0$ instead, the probability of "beautiful" would be lower, and "rainy" and "cold" would have had more of a fighting chance. The temperature $T = 0.7$ has sharpened the distribution, amplifying the model's confidence in "beautiful" beyond what the raw logits at $T = 1.0$ would have implied. Notice that "enormous" has only a 0.3% chance: semantically, "The weather in Paris is enormous" is nonsensical, and the model's logits correctly reflect this, assigning a strongly negative logit. Even at $T = 2.0$ , the probability of "enormous" would only rise to about 2%: a sensible result.

Let's confirm these calculations with code:

In[17]:

Code

# Numerical trace of temperature scaling
example_tokens_trace = ["beautiful", "rainy", "cold", "unusual", "enormous"]
example_logits_trace = torch.tensor([3.5, 2.0, 1.8, 0.9, -0.5])
T_trace = 0.7

Out[18]:

Console

Step 1: Scaled logits (z_i / T):
  beautiful   :   3.50 / 0.7 =   5.000
  rainy       :   2.00 / 0.7 =   2.857
  cold        :   1.80 / 0.7 =   2.571
  unusual     :   0.90 / 0.7 =   1.286
  enormous    :  -0.50 / 0.7 =  -0.714

Step 2: Exponentiated (exp(z_i / T)):
  beautiful   :  148.413
  rainy       :   17.412
  cold        :   13.085
  unusual     :    3.617
  enormous    :    0.490

Step 3: Normalization constant Z = 183.016

Step 4: Final probabilities:
  beautiful   : 0.8109 (81.1%)
  rainy       : 0.0951 (9.5%)
  cold        : 0.0715 (7.1%)
  unusual     : 0.0198 (2.0%)
  enormous    : 0.0027 (0.3%)

Step 5: Sum check = 1.000000

The computed values confirm our hand calculation. Notice how dramatically the temperature has concentrated probability on "beautiful": it captures over 81% of the probability mass despite its logit being only 1.5 units above the second-highest token. The exponential nature of softmax magnifies even moderate logit differences into large probability gaps.