Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Multi-Server and Deployment Factors

## Different Instances or Shards Typically, cloud providers run LLMs on clusters of machines. Your request might go to one of many servers. If not every server is exactly identical (same hardware, same software version, same numeric libraries), their outputs could vary slightly. Even if the provider tries to keep them consistent, subtle differences (one machine has a newer GPU driver, or a slightly different GPU model, etc.) can introduce variation. For example, one server might process using an A100 GPU and another an A10 GPU - as discussed earlier, those can produce slight differences in results. So if you call the API at two different times, behind the scenes you might be hitting different hardware or a different partition of the model. ## Model Updates and Configuration Changes AI providers frequently update models or adjust their settings. OpenAI has acknowledged that. Even when you try to make the API calls deterministic, changes on OpenAI's end can impact outputs. Even with the same seed and parameters set that were meant to give repeatable results across calls, they caution that this is only “mostly” deterministic. ## Load Balancing and Batching As touched on earlier, if the service batches multiple user requests together to maximize throughput, it can lead to cross-influence (especially in MoE models). Even without MoE, batching can affect low-level numerical behavior. For example, frameworks might pad sequences in a batch or handle memory differently when batch size changes, which can shift the arithmetic slightly. If on one API call your prompt is processed alone, and on another it happens to be processed alongside 3 other prompts (to better utilize the GPU), the internal execution differs. This can contribute to output differences. ## Sharding Large Models Very large models (like GPT-4) might be too big for one GPU, so they are sharded across multiple GPUs or machines. This means each neural net forward pass involves communication between parts of the model running on different devices. If timing or communication patterns vary, it could introduce slight nondeterminism as well. For example, if two GPUs return results in a different order or with minor timing differences, the aggregator might produce non-bit-identical outcomes. These effects are usually small, but over many tokens, again, could show up as a difference in one token choice.