Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Internal Model Architecture and Framework Factors

## Mixture-of-Experts (MoE) Layers Some large models (most likely GPT-4) use a Mixture of Experts architecture, where the model has multiple expert sub-networks and a routing mechanism decides which “expert” handles a given token. This can introduce nondeterminism in an interesting way. For efficiency, the model might process multiple input queries together in a batch. When it does, tokens from different sequences could compete for the same expert resources (because typically only a limited number of tokens can go to each expert). If two requests are handled together, they might interfere with each other's expert routing. As one analysis noted, in sparse MoE systems. This means the output for a single query might differ depending on what other queries were processed alongside it. If you send the same prompt twice but behind the scenes it was batched with different other user requests each time, an MoE-based model could give different answers. This is a unique source of nondeterminism that comes purely from the model's architecture and how the service batches workloads, rather than from the usual sampling logic. ## Non-deterministic Operations in Frameworks The libraries and frameworks used to run LLMs (such as TensorFlow, PyTorch, CUDA libraries, etc.) sometimes use optimized algorithms that are nondeterministic. For example, certain GPU operations (like some matrix decompositions, parallel reductions, or convolution algorithms) might trade a bit of determinism for speed. These ops might spawn threads or use atomic operations where the order of execution isn't fixed, leading to slight run-to-run differences. The PyTorch documentation, for instance, has a whole reproducibility guide detailing which operations are not deterministic by default and how to avoid them. If an LLM uses any of these under-the-hood operations, you could see variations unless special care is taken. Generally, deep learning frameworks do offer settings to enforce determinism (e.g., flags to use deterministic algorithms for certain operations), but these often come at a performance cost and are not always enabled in production systems. ## Software Bugs or Undefined Behavior It's also worth acknowledging that sometimes nondeterminism can come from simple bugs or from parts of the code that aren't intended to be random but lack guarantees. For example, if the model relies on some external state or if there is a race condition in how outputs are assembled, it could manifest as unpredictable output differences. In early discussions, even OpenAI engineers were puzzled by temperature-0 variability and wondered if it was a bug or floating-point issue. Over time, the community has largely converged on the understanding that it's mostly due to the factors listed (floating point and parallelism), but the point is that the entire software stack needs to be deterministic to get identical outputs - and that's a hard thing to achieve in distributed, optimized systems.