Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Parallelism and Hardware Variability

Large language models are usually run on GPUs or TPUs that perform many operations in parallel for speed. Parallel computation can introduce nondeterminism because the order in which operations occur can vary. Unlike a simple calculator that adds numbers in a fixed sequence, a GPU might add up many numbers simultaneously in different chunks, then combine the results. Due to the way floating-point addition is not perfectly associative (i.e. (a + b) + c may not equal a + (b + c) when rounding is involved), doing operations in a different order can lead to slight differences in the final sum. The model's probability calculations might therefore differ slightly from run to run depending on how the work is partitioned and parallelized. Hardware differences also play a role. If you run the same model on two different GPU models (say an Nvidia Tesla T4 vs. an A100), you might not get byte-for-byte identical results. Different processors have different architectures and optimization routines for math, which can produce minutely different results. One experimental report showed that the exact same prompt on two GPU types yielded slightly different token probabilities, causing the text generation to diverge after a few words. In most cases, these differences are tiny and don't change the highest-ranked token, so you wouldn't notice any change, but once in a while, especially in a long output, the difference can affect which token is deemed “most likely” at a given step. This is why LLM outputs can differ across runs or machines purely due to low-level hardware and parallelism details.