Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Decoding Strategies and Tie-Breaking

Even when no randomness is intended, the decoding algorithm (the procedure that picks tokens) can introduce variability. Temperature is one factor in decoding, but it's often used alongside others like top-k or top-p (nucleus) sampling, or beam search. At temperature 0, most applications switch to a greedy selection (always pick the top token). However, if any other sampling parameters are still in play, they can affect determinism. For example, if top_k is set greater than 1 or top_p is less than 1, the model is still allowed to consider multiple tokens for each position, which can reintroduce randomness. To truly force determinism, you typically set top_k = 1 (only one option considered) and top_p = 1 (no probability mass cutoff) - or, equivalently, top_p = 0 in some implementations, to only allow the single top token. If these aren't set strictly, the model might still do a form of sampling among the top candidates, causing variations in output. In short, using temperature=0 alone isn't always sufficient if other decoding parameters allow choice. Another issue arises when two or more tokens have nearly equal probability (a tie or close to a tie). In theory, if one token's probability is 0.5000 and another's is 0.4999, greedy decoding will pick the 0.5000 token. But imagine due to rounding or minor differences, sometimes the model sees the second token as 0.5001 vs 0.4999 for the first. Now the “winner” flips. Even with a temperature of 0, the model needs a way to break ties or decide when probabilities are equal or extremely close. Different implementations might handle this differently: one might consistently pick the token that comes first alphabetically or by ID number; another might introduce a tiny random jitter to break ties. If the tie-breaking rule isn't consistent, you could occasionally get a different outcome. For example, OpenAI's documentation for some models notes that if you set temperature to 0, the system will “automatically increase the temperature until certain thresholds are hit”. This suggests that the API may raise the temperature slightly in cases where it needs to make a decision, effectively introducing a bit of randomness to avoid deadlocks or repetitive loops. So, what you thought was strict greedy mode might, under the hood, allow a tiny amount of flexibility, just to ensure the model continues generating. Additionally, if using beam search (a deterministic search algorithm that finds a high-probability completion), you might expect it to be reproducible. Beam search is mostly deterministic, but if multiple beams have exactly the same score, the tie might be broken arbitrarily (or by nondeterministic ordering of floating-point sums, as discussed). This is usually rare, but it's another edge case where nondeterminism can slip in.