Checkpointing and Recovery › Checkpointing and Recovery

Training a large language model takes weeks. A single job might run for twenty days, consuming thousands of GPU-hours and hundreds of thousands of dollars in compute. What happens when a GPU fails on day seventeen? Without a checkpoint, you restart from scratch. With a well-designed checkpointing strategy, you resume from a few hours ago at most, losing only a small fraction of your investment.

Checkpointing is the practice of periodically saving the complete state of a training run to durable storage, so that when something goes wrong (and in distributed training, something always goes wrong) you can restore that state and continue as if the interruption never happened. It sounds simple, but the implementation raises several questions: what exactly must you save, how often should you save it, how do you avoid stalling 512 GPUs while writing 200 GB to disk, and how do you ensure the recovered run produces the same results as the uninterrupted run would have?

The answer to that last question is worth lingering on. Exact reproducibility matters more than it might initially seem. During a long research run, you may need to debug a training divergence, verify that two experimental variants behaved identically up to a given step, or produce evidence for a paper that a particular result was obtained by a specific training procedure. A checkpointing strategy that recovers training in a way that looks approximately correct is not the same as one that recovers it exactly. The two goals require saving different state, and the cheaper approximation will eventually cost you.

This chapter covers checkpoint contents in depth, the tradeoffs behind checkpoint frequency decisions, asynchronous checkpointing to hide I/O latency, and the full fault recovery workflow that gets a distributed job back on its feet after a failure. It also covers the practical operational details that textbooks often skip: how to detect failures quickly, how to handle resharding when you recover onto different hardware, and how to defend against silent data corruption.

Building on the distributed training concepts examined in the data parallelism, tensor parallelism, and FSDP chapters, you now understand that training state is spread across many devices. Checkpointing must collect all of that distributed state into a coherent, recoverable snapshot. The next chapter covers monitoring and observability, which works hand-in-hand with checkpointing to detect failures quickly and trigger recovery.