Checkpointing and Recovery › Limitations and Practical Considerations

Checkpointing solves the fault tolerance problem elegantly at the conceptual level, but several real-world limitations shape how it is deployed in practice. Understanding these limitations helps you design a checkpointing strategy that works under production conditions rather than just in theory.

Large model checkpoints are expensive to store and transfer. A 70B model with full optimizer state requires roughly 420 GB per checkpoint. At cloud storage prices (approximately $0.02 per GB-month for standard storage), retaining 100 such checkpoints costs over $840 per month just for the storage. With fast object storage tiers, the per-GB price can be several times higher. Organizations must balance the safety of frequent checkpoints against their storage budgets, which often leads to aggressive rolling policies that minimize historical retention.

Storage bandwidth is equally constraining. Writing 420 GB at 10 GB/s takes 42 seconds. Even with asynchronous I/O, if your checkpoint interval is 10 minutes and your write takes 42 seconds, you need the async I/O to complete before the next checkpoint is triggered. For very large models or slow storage systems, you may need to extend the checkpoint interval, accept a larger I/O overhead, or invest in faster parallel storage. Some organizations solve this by provisioning high-bandwidth local NVMe storage on each training node, achieving 30-50 GB/s per node and dramatically shortening checkpoint write times.

The choice of checkpoint format also affects bandwidth requirements. PyTorch's torch.save() uses pickle serialization, which is flexible but not optimally efficient for large tensors. The safetensors format, developed specifically for large model weights, uses a simple binary layout that can be memory-mapped directly, enabling faster loads and more efficient storage. For very large models at scale, switching from torch.save() to safetensors can reduce both checkpoint write time and load time significantly.

Silent data corruption (SDC) is a failure mode where computation produces incorrect results without any obvious error signal. The model's loss may continue to decrease normally, but the corrupted values spread through the model parameters over subsequent steps. By the time the corruption is detected (often through downstream evaluation on held-out tasks), it may have propagated through hundreds of checkpoints and thousands of training steps.

SDC is more common on commodity GPU hardware than on ECC-protected memory, and its frequency increases with model size and training duration. Major LLM training efforts at companies like Google and Meta have reported SDC events that required careful forensic analysis to trace back to specific hardware faults. The events ranged from subtle bit flips that produced slightly wrong gradient values to more dramatic faults that corrupted entire parameter tensors.

The main defense against SDC is maintaining a long-term archive of permanent checkpoints with sufficient spacing to provide a clean restore point before the corruption began. Some teams also run periodic consistency checks: saving a "shadow" checkpoint and verifying that the model's forward pass produces the expected outputs on a fixed validation batch. If the forward pass loss drifts unexpectedly from the historical baseline, the consistency check triggers an alert before too much corrupted training has accumulated.

A complementary defense is gradient anomaly detection: tracking gradient norms per layer and alerting when any layer's norm exceeds a threshold multiple of its recent historical average. A single SDC event often produces an anomalously large gradient in the affected layer during the step when the corruption first affects the computation. Catching this in real time, before the next checkpoint is written, prevents the corrupt state from ever entering the checkpoint.

In distributed training, all ranks must checkpoint at exactly the same step. If rank 0 checkpoints at step 1,000 but rank 1 checkpoints at step 1,001 (due to a scheduling discrepancy or a clock skew between nodes), the combined checkpoint is inconsistent: the model parameters and optimizer state from the two ranks are out of sync. Loading such a checkpoint and running a forward pass would produce different results on different ranks, causing collective operations (like AllReduce for gradient synchronization) to hang or produce garbage values.

Ensuring consistency requires a barrier at checkpoint time: all ranks must complete their writes before the "latest checkpoint" pointer is updated. PyTorch's distributed checkpoint library manages this through a coordinator rank that only marks the checkpoint complete after receiving acknowledgment from all participant ranks. The barrier is implemented using a distributed rendezvous operation that guarantees all ranks have committed their shard before any rank proceeds.

This consistency requirement interacts with the asynchronous checkpointing design. In the asynchronous case, the training barrier is placed after the fast serialization phase (GPU-to-CPU copy), not after the slow write phase. All ranks finish their serialization and collectively confirm they are ready before any rank launches its background writer. This ensures the checkpoint represents a consistent model state, even though the background writes proceed independently and at potentially different speeds.

In production, the recovery time from failure to resumed training matters as much as the checkpoint interval. If checkpointing is frequent but loading is slow (for example, loading a 420 GB checkpoint over a slow network), the total downtime per failure can be significant even when very little training was lost. For a checkpoint loading at 5 GB/s, a 420 GB checkpoint takes 84 seconds just for I/O. Add process startup time (tens of seconds for a large job), initialization, data pipeline warmup, and verification, and recovery can take 5-10 minutes even with a checkpoint from 15 minutes ago.

For a cluster with a 6-minute MTBF (10,000 GPUs at 1,000 hours MTBF per device), spending 5-10 minutes recovering from each failure means the cluster spends nearly as much time recovering as it does training. Reducing recovery time is as important as reducing checkpoint interval.

Pre-fetching checkpoints to local storage, using high-bandwidth interconnects for checkpoint loading, and structuring checkpoints as smaller sharded files that load in parallel are all techniques used to minimize recovery time. Some organizations maintain a local SSD cache on each training node: the most recent checkpoint is always available locally, allowing fast recovery from node-local failures and shortcutting the network bottleneck for other failure types.

The recovery time also depends on how quickly failures are detected. As discussed in the fault detection section, NCCL's default 30-minute timeout means that the total outage time for a single GPU failure is 30 minutes (detection) plus 5-10 minutes (recovery), totaling 35-40 minutes even with a 15-minute checkpoint interval. Faster detection dramatically reduces total downtime per failure, often more than any optimization to the checkpoint format or storage infrastructure.