Catastrophic Forgetting › Summary

Catastrophic forgetting occurs when fine-tuning overwrites the neural representations responsible for pre-trained capabilities. This phenomenon stems from the basic stability-plasticity dilemma: neural networks must be plastic enough to learn new tasks but stable enough to retain old knowledge. Standard gradient descent optimizes only for the new task and provides no signal to preserve prior capabilities, making forgetting an inevitable consequence of naive fine-tuning.

At the parameter level, forgetting manifests as drift of high-importance weights away from their pre-trained values. Not all parameters are equally affected: weights necessary for prior capabilities suffer the most when fine-tuning gradients conflict with the gradient directions that would maintain those capabilities. Gradient interference, where the fine-tuning gradient opposes the direction needed to preserve prior performance, is the primary mechanism through which this conflict causes damage.

Measuring forgetting requires evaluating models on held-out tasks representing prior capabilities, using metrics like backward transfer, the forgetting measure, or benchmark suite performance. Without measurement, forgetting goes undetected until deployment reveals unexpected failures. The forgetting measure captures worst-case degradation from peak performance, while backward transfer captures the average change from initial learning, and perplexity on diverse text provides a task-agnostic probe of general language modeling competence.

Mitigation strategies span multiple approaches. Regularization methods like L2-SP and EWC penalize parameter drift from pre-trained values by adding soft constraints to the optimization objective. Replay methods mix prior task data into fine-tuning batches, maintaining training signal for prior capabilities alongside the new task. Architecture methods freeze original weights while adding trainable components, achieving the strongest forgetting prevention at the cost of reduced adaptation flexibility. Learning rate strategies apply differential updates across layers, protecting general features in early layers while letting rapid adaptation in later layers.

Preserving pre-trained capabilities requires combining these techniques with careful monitoring. Layer-wise learning rates protect general features in early layers. Gradual unfreezing allows controlled adaptation that starts with the most volatile parameters frozen and progressively opens up the model. Evaluation-driven stopping provides a direct safety mechanism, halting training as soon as forgetting exceeds an acceptable threshold.

As we'll see in the Fine-tuning Learning Rates chapter, the choice of learning rate schedule strongly affects both task learning and forgetting dynamics. The PEFT methods covered later add small trainable modules while keeping pre-trained weights completely frozen, effectively eliminating forgetting at the cost of reduced adaptation flexibility.