Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Temperature = 0: Theoretical Determinism vs. Reality

<TLDR> Temperature = 0 in language models aims to eliminate randomness by always selecting the most probable next token (i.e., greedy decoding). Despite this, real-world LLMs still produce varying outputs due to factors like: - Floating-point precision limitations - Hardware and parallel processing variability - Decoding tie-breakers - Other sampling parameters like top-k or top-p that may still introduce variation Additional nondeterminism comes from model and infrastructure complexity, including: - Mixture-of-Experts (MoE) architectures - Nondeterministic operations in frameworks like PyTorch or TensorFlow - Multi-GPU sharding - Batching, load balancing, and cross-server variability While temperature=0 increases consistency, true repeatability requires strict control over hardware, software, and decoding settings. Ultimately, these nuances explain why outputs can still differ, even when no randomness is explicitly introduced. </TLDR> By now, most of us know that LLMs are not entirely deterministic, meaning that given the same input, they will not always produce the same output. This is the case even when we set the temperature to 0. This is usually a setting that is only available to engineers, but conceptually, it means that we are dialing the randomness down to zero. But if there's no randomness anymore, why is it still not perfectly deterministic? I've asked this question to multiple experts and never got a compelling or clear answer, so I did some research to answer it. To recap, in theory, temperature 0 means the model always picks the single most probable next token at each step. This is like a greedy strategy: one token has a probability of 1 (100%) and all others are 0, so there is only one choice at each step. In an ideal world, that would guarantee the exact same completion every time. In practice, however, LLM outputs can still vary slightly even at temperature 0, due to several technical factors. What does “temperature 0” mean? In text generation, temperature is a parameter that controls randomness. A higher temperature (e.g., 0.8 or 1.0) makes the model more creative or random, while a lower temperature (e.g., 0.2) makes it more focused and conservative. Mathematically, temperature scales the model's confidence distribution (via the softmax function). As temperature approaches 0, the softmax “peaks” more sharply on the highest-probability token, essentially making that token the only viable choice. In fact, setting T = 0 is treated as a special case - it's like telling the model to always pick the top-scoring token at each step (often called greedy decoding). In theory, this should remove randomness entirely: if there is only one choice at each step, the sequence of choices should be the same every time. Why do we still see different outputs sometimes? The answer is that the real-world implementation of LLMs involves many subtle complexities. Setting temperature to 0 does increase determinism, but it does not guarantee perfect repeatability. Imagine two people performing very long and complex calculations by hand. Even if they follow the same steps, tiny differences (like rounding a number differently) could lead to a different final result. Similarly, inside an LLM's generation process, there are tiny numerical and procedural variations that can lead to divergent outputs. Below, we break down the key factors that contribute to non-deterministic behavior even when no randomness is intentionally introduced.