Checkpointing and Recovery › Checkpoint Frequency

How often should you checkpoint? The answer depends on three competing factors: the cost of lost work if a failure occurs, the overhead of writing the checkpoint, and the storage cost of retaining multiple checkpoints. These three factors pull in opposite directions, and the optimal policy depends on your specific infrastructure.

Suppose your training run takes $T$ total hours and you checkpoint every $\Delta$ hours. If a failure occurs at a uniformly random time between two checkpoint events, the expected work lost is $\Delta / 2$ hours. Over the entire run, if there are $F$ failures, the expected total work lost is $F \cdot \Delta / 2$ hours. Since longer runs experience more failures, the expected fraction of total work lost scales with $\Delta$ and the failure rate.

For production-scale runs, the mean time between failures (MTBF) for individual GPUs or network switches is on the order of months, but a cluster of thousands of devices will experience failures far more frequently. The failures are not limited to GPU hardware faults: network switches fail, cooling systems fail, storage systems become unavailable, and job schedulers sometimes preempt running jobs. Each of these events terminates the training job and requires recovery.

With 1,000 GPUs each having an MTBF of 1,000 hours, the cluster as a whole has an effective MTBF of just 1 hour. With 10,000 GPUs, it drops to 6 minutes. Checkpointing every 10-15 minutes is common for very large runs precisely because of this arithmetic.

Formally, if the failure rate per device is $\lambda$ (failures per hour) and you have $N$ devices, the cluster failure rate is approximately $N\lambda$, assuming independent device failures. The expected time until the first failure is:

$$\mathbb{E}[\text{time to first failure}] = \frac{1}{N \lambda}$$

where:

• $N$: number of devices in the cluster
• $\lambda$: per-device failure rate (failures per unit time, the reciprocal of MTBF)
• $N\lambda$: cluster-wide failure rate, assuming independent failures

For a sensible checkpoint interval $\Delta$, you want:

$$\Delta \ll \frac{1}{N \lambda}$$

so that the expected fraction of work lost per failure is small. As $N$ grows, $1/(N\lambda)$ shrinks, requiring shorter checkpoint intervals to maintain the same expected loss per failure. The implication is stark: a policy that worked well for a 100-GPU run will be inadequate for a 10,000-GPU run at the same failure rate per device, even if the model is the same size.

Failures are not always uniformly distributed in time. Hardware failures cluster after initial deployment (infant mortality), during thermal stress events, and during firmware updates. Practical checkpoint intervals should be conservative enough to handle bursts of failures during these higher-risk periods, not just average-rate failures.

Checkpointing is not free. Writing a 300 GB checkpoint to NFS or cloud object storage at 10 GB/s takes 30 seconds. If you checkpoint every 10 minutes, you spend 30/600 = 5% of compute time blocked on I/O. At 1%, this overhead is acceptable and generally goes unnoticed. At 10%, it becomes a meaningful training slowdown that shortens the effective training budget.

Synchronous checkpointing, where all GPU work pauses while the checkpoint is written, is the simplest approach but has the highest overhead. Asynchronous checkpointing, described in the next section, overlaps I/O with computation to recover most of this overhead. But even with asynchronous I/O, the serialization phase (copying tensors from GPU memory to CPU memory) blocks training briefly. For a 300 GB checkpoint at 60 GB/s PCIe bandwidth, the serialization alone takes about 5 seconds.

The I/O model also interacts with distributed training. In a model-parallel or FSDP setup where parameters are sharded, each rank writes only its shard. If you have 64 ranks each writing 5 GB to separate storage volumes with 10 GB/s bandwidth each, the parallel write takes 0.5 seconds, far better than having a single rank write 320 GB sequentially. The storage infrastructure therefore determines the effective checkpoint bandwidth and directly influences how frequently you can checkpoint without unacceptable overhead.

Retaining every checkpoint would be prohibitively expensive. A 70B model in BF16 with full Adam state requires roughly 420 GB per checkpoint. If you checkpoint every 10 minutes over a 20-day run, that is 2,880 checkpoints totaling over 1.2 petabytes of storage.

The practical solution is a rolling checkpoint policy: retain only the last $K$ checkpoints, deleting older ones as new ones arrive. $K = 3$ to $K = 5$ is typical for operational recovery. With a 15-minute checkpoint interval, this gives you 45-75 minutes of history, enough to recover from virtually any transient hardware failure.

Beyond the rolling window, many teams also retain a sparse set of permanent checkpoints at regular intervals (every 1,000 steps, for example) for research purposes. These allow comparison at different training stages, ablation studies, and emergency fallback if a silent data corruption is discovered days after it occurred. Permanent checkpoints are a small fraction of all checkpoints but provide a safety net that rolling-only policies cannot provide.

Silent data corruption is a real concern and motivates the permanent checkpoint policy. GPUs occasionally produce bit-flip errors in computation results without any hardware exception. The error may not cause an obvious crash but instead slowly degrades the model, producing weights that are slightly wrong and growing slightly more wrong with each contaminated gradient update. If you discover this 12 hours after it began, you need a checkpoint from before the corruption. A retention policy with $K = 5$ at 15-minute intervals only gives you 75 minutes of history, which is not enough when the corruption has been propagating for half a day.

The typical approach is two-tier retention: rolling checkpoints for operational recovery (last $K$, deleted aggressively), and permanent checkpoints at milestone steps for long-term fallback. Some organizations add a third tier: archival checkpoints kept indefinitely for models that reach certain loss thresholds or evaluation benchmarks. These represent training milestones worth preserving regardless of whether they are needed for recovery.