Checkpointing and Recovery › Summary

Checkpointing is the fault tolerance mechanism that makes long distributed training runs practical. A complete checkpoint must contain the model parameters and optimizer state, plus the scheduler state, gradient scaler, RNG state, and data loader position. Missing any of these can cause the resumed run to diverge subtly from the original, corrupting research reproducibility and potentially wasting hundreds of GPU-hours before the problem is noticed.

Checkpoint frequency is governed by the tradeoff between expected work lost per failure and I/O overhead. Large clusters fail frequently, requiring frequent checkpoints, but large models are expensive to checkpoint. For a 10,000-GPU cluster with per-device MTBF of 1,000 hours, the expected time between cluster-level failures is just 6 minutes, demanding checkpoint intervals of that order. Synchronous checkpointing at such intervals would be completely infeasible for large models.

Asynchronous checkpointing resolves this tension by overlapping disk I/O with GPU computation. The training stall is reduced from the full checkpoint write time (tens of seconds) to just the GPU-to-CPU copy time (a few seconds), recovering the majority of the checkpointing overhead. The trade-off is an increased CPU memory requirement: you need space for two full copies of the model state simultaneously.

Rolling checkpoint policies balance recoverability against storage cost by retaining only the last $K$ checkpoints while preserving permanent checkpoints at sparser intervals for long-term fallback. The two-tier policy provides operational recovery for transient hardware failures and research-grade recovery for silent data corruptions that may not be detected for days.

Fault recovery involves detecting the failure, identifying the latest valid checkpoint, restarting the distributed job (possibly with resharding if hardware has changed), loading and restoring all state, and verifying that the resumed run matches expectations before continuing. With well-designed checkpointing and fast failure detection, a training job that loses a GPU on day seventeen can resume within minutes from a checkpoint taken at most 15 minutes earlier, losing only a tiny fraction of the total compute investment.

The next chapter on monitoring and observability covers how to detect training anomalies in real time, which works alongside checkpointing to catch problems quickly and determine exactly when and why a training run deviated from expected behavior.