Replay Methods › Summary

Replay methods address catastrophic forgetting by making the training distribution over all tasks visible to the model even as new tasks arrive sequentially. The core idea: if forgetting happens because the model stops seeing old data, the fix is to keep showing it old data, either directly through stored examples or indirectly through synthetic reconstructions.

The key ideas from this chapter are:

• Experience replay stores raw past examples in a buffer and mixes them into training batches for new tasks. Reservoir sampling ensures unbiased coverage of all past tasks. Selection strategies (herding, gradient-based, uncertainty-based) optimize which examples to store for maximum protective effect per slot.

• Dark Experience Replay extends experience replay by storing the model's logit outputs alongside each saved example. Training the model to reproduce its previous logit distributions, rather than just correct labels, preserves richer representation structure and more effectively prevents forgetting with the same buffer budget.

• Pseudo-rehearsal generates synthetic proxy inputs and uses the model's own outputs as training targets, preserving old input-output mappings without storing any real data. Random inputs are cheap and simple but fail to span the true input distribution in high-dimensional spaces, limiting their effectiveness.

• Generative replay trains a generative model (VAE or GAN) on each task and uses it to produce realistic synthetic examples during future training. The dual-memory architecture parallels hippocampal-neocortical complementary learning. However, the generative model itself is subject to forgetting and introduces compounding generation quality loss across many task transitions.

• Replay selection matters significantly for small buffers. Herding selection outperforms random sampling in class-incremental settings, while uncertainty-based selection is more effective in domain-incremental settings. The optimal strategy depends on the task structure and the continual learning scenario.

• LLM replay takes the form of data mixing: including a fraction of pretraining or earlier fine-tuning data in subsequent fine-tuning runs. This is experience replay at scale, and the mixing fraction is one of the most important hyperparameters for preventing capability degradation in fine-tuning pipelines.

The choice between experience replay, pseudo-rehearsal, and generative replay depends on your constraints. If you can store data and have no privacy restrictions, experience replay with careful selection is the most reliable option. If data retention is impossible, generative replay offers a principled alternative at the cost of generative model complexity and the risk of compounding generation errors. If neither memory nor compute budget allows for sophisticated replay, pseudo-rehearsal provides a lightweight fallback with partial forgetting protection. The right answer is almost always some form of replay. The question is which form your deployment constraints permit.