Replay Methods › Replay Methods

In the previous chapters, we explored the core problem of catastrophic forgetting and how regularization methods like Elastic Weight Consolidation (EWC) address it by penalizing changes to parameters that were important for earlier tasks. Regularization works by constraining the learning process from the inside: it shapes the loss landscape so that gradient descent naturally avoids overwriting critical knowledge. The approach is elegant, but it faces a fundamental scaling challenge. The number of constraints needed to protect $k$ tasks grows with $k$, and the constraints imposed by early tasks can conflict with the plasticity demanded by later ones.

Replay methods take a completely different approach. Rather than constraining how the model learns, they influence what the model learns from. The central idea is simple: if the model forgets old tasks because it never sees data from them again, the obvious fix is to keep showing it that data. Replay methods store or reconstruct examples from past tasks and mix them into the training stream for new tasks, so the model always faces a joint learning problem even when the incoming data stream is sequential. From the optimizer's perspective, each gradient update receives a signal from multiple task distributions simultaneously. There is no forgetting because there is no exclusive focus on any single task.

This intuition connects directly to how biological memory works. The hippocampus is thought to replay neural activity patterns from waking experience during sleep, allowing the neocortex to consolidate memories without catastrophic interference. Computational replay methods are a direct analogue: a memory system stores or generates past experiences, which are periodically replayed to maintain previously learned representations. The theoretical framework of complementary learning systems, developed in cognitive neuroscience to explain why both fast episodic memory (hippocampus) and slow semantic memory (neocortex) are necessary for stable generalization, maps almost exactly onto the architecture of generative replay systems. The hippocampal fast learner captures specific episodes; the neocortical slow learner integrates them into structured knowledge. Replay methods recreate this two-system structure in software.

The replay family contains three main branches. Experience replay stores actual data samples in a buffer and retrieves them during training. Pseudo-rehearsal generates synthetic proxy data to approximate past distributions, avoiding the need to store real examples. Generative replay uses a learned generative model to produce realistic reconstructions of past data on demand. Each branch comes with its own design choices, practical constraints, and failure modes, and together they form the most widely used family of continual learning methods in both research and practice.

Understanding why replay is so effective requires identifying what catastrophic forgetting destroys. When the model trains exclusively on task $T_2$, the gradient signal from that task dominates every parameter update. The loss landscape for task $T_1$ is invisible to the optimizer, so the parameters drift freely in directions that may erase task $T_1$'s knowledge entirely. Replay restores visibility: by including task $T_1$ examples in each batch, the optimizer sees a gradient signal that is a sum over both task distributions. The parameter trajectory is pulled simultaneously toward both tasks' optima, and catastrophic forgetting is replaced by the gentler phenomenon of multi-task interference, where performance on each individual task is somewhat lower than it would be with exclusive training, but no task is catastrophically erased.