Why Temperature=0 Doesn't Guarantee Determinism in LLMs › Floating-Point Precision Limitations

One fundamental issue is that computers do not have infinite precision when handling numbers. LLMs rely on many mathematical operations with probabilities (or logits, the raw scores for tokens) represented in floating-point format. Floating-point numbers are an approximation; they have limited decimal precision. This means rounding errors can occur at many steps, and these tiny errors can cascade through the model's calculations. Over the course of generating a long piece of text, minute differences in arithmetic can add up to a different outcome.