Replay Methods › Comparing Replay Approaches

The performance gap between herding-based experience replay and the no-replay baseline illustrates why replay has become the dominant continual learning paradigm in practice. Herding selection retains the most information per buffer slot by choosing representative examples, while the no-replay baseline falls below 20% after five tasks. Generative replay performs similarly to random experience replay, which is encouraging given that it requires no stored data: the cost of compressing task distributions into generative model parameters is roughly equivalent to the cost of storing a small but well-curated random sample.

Pseudo-rehearsal occupies a lower tier in this comparison. Its forgetting protection is consistent across tasks but weaker than methods that retain information from the true distribution. This reflects the fundamental limitation identified earlier: random inputs do not span the true input distribution, so the rehearsal signal does not protect the most important regions of the input-output mapping.

Experience replay grows memory linearly with the number of tasks and buffer size. With $k$ tasks and a buffer of $C$ examples, storage cost is $O(C \cdot d_{\text{input}})$, where $d_{\text{input}}$ is the input dimensionality. For text tasks with long sequences, this can be substantial. For image tasks at typical benchmark resolutions (32x32 to 224x224), even buffers of 10,000 examples are manageable on modern hardware.

Generative replay trades memory for compute. Instead of storing raw examples, it stores generative model parameters. For large datasets, this can be much cheaper: a GAN trained on 100,000 images might have 10M parameters (40 MB at float32), while storing even 5,000 raw images at 224x224 pixels requires similar storage. The key cost is computational: generating replay batches requires forward passes through the generative model at every training step.

Pseudo-rehearsal has negligible memory and compute overhead beyond the base model, but this comes at the cost of replay quality. Random inputs may not effectively protect the decision boundary or class structure of past tasks, especially in high-dimensional spaces.

Replay methods behave differently across the three standard continual learning settings, and understanding these differences is important for applying them correctly.

In the task-incremental setting, the model has access to the task identity at inference time. A separate classification head is typically maintained for each task, and the model simply routes each test example to the correct head based on the provided task label. In this setting, forgetting is almost entirely a representation learning problem: the shared feature extractor must maintain useful representations for all tasks, but the task-specific heads protect against direct output interference. Replay helps maintain the quality of shared representations, and relatively small buffers are sufficient.

In the class-incremental setting, the model must classify among all classes seen so far without knowing which task the test example came from. This is substantially harder. The classification head must grow as new classes are added, and it must simultaneously distinguish between classes from all tasks. Here, replay is critical for representations and for the classifier boundary. Without replay of past task examples, the growing classification head will dominate the loss during new task training, causing the features and decision boundaries for past classes to degrade. Herding-based replay was specifically designed for this setting, where maintaining accurate class mean representations is key to the nearest-class-mean classification approach used in iCaRL.

In the domain-incremental setting, the output structure remains fixed but the input distribution shifts across tasks. For example, a sentiment classifier first trained on movie reviews and then adapted to product reviews must maintain performance on both domains with the same output classes. Replay in this setting focuses on anchoring the input-output mapping in the original domain's distribution, making uncertainty-based selection (which prioritizes boundary examples) particularly effective.

The standard Split-CIFAR and Permuted-MNIST benchmarks that appear throughout the continual learning literature each test a different variant of these settings. Split-CIFAR, where CIFAR-10 is divided into 5 two-class tasks presented sequentially, most closely approximates the class-incremental scenario with the same output space. Permuted-MNIST, where each task applies a different fixed permutation to the input pixels, approximates the domain-incremental scenario.

The replay intuitions developed for smaller classification models become more complex when applied to large language models. LLMs present several unique challenges that require adaptations to the standard replay framework.

Dataset scale: Pre-training datasets for LLMs contain trillions of tokens. Storing even a tiny fraction of this data for replay is prohibitively expensive. In practice, replay for LLMs uses document-level examples rather than individual training instances, and buffer capacity is measured in millions rather than thousands of examples. The per-example cost is also higher: a single long-context document may contain thousands of tokens, making each replay example significantly more memory-intensive than a single image or feature vector.

Implicit replay through data mixing: Many LLM training pipelines include replay implicitly. When fine-tuning a language model on a new domain, practitioners often mix in a fraction of pre-training data to maintain general language understanding. This is experience replay at scale, though it is typically not framed that way in the literature. The mixing ratio, the pre-training data selection strategy, and the curriculum order are all forms of replay hyperparameters, even if they are not explicitly identified as such.

Instruction tuning and RLHF: When fine-tuning with instruction data or reinforcement learning from human feedback, replay of base pre-training examples or earlier fine-tuning data is commonly used to prevent the model from losing its general capabilities. The replay fraction is a critical hyperparameter: too little and the model forgets; too much and it fails to acquire the new behavior. Studies on instruction-tuned models have found that including even 5-10% general pre-training data in the fine-tuning mix substantially preserves general capability while allowing the model to acquire task-specific behavior.

Catastrophic forgetting in fine-tuning: The forgetting problem becomes especially visible when fine-tuning on narrow domains. A model fine-tuned extensively on medical texts will lose its ability to discuss other topics, answer general knowledge questions, and maintain coherent multi-turn conversations. Mixing a small amount of general web data into the fine-tuning corpus is a practical replay strategy that significantly mitigates this degradation at low cost. The selection of which web documents to include can itself be optimized: documents that are semantically distant from the fine-tuning domain provide the most complementary signal for maintaining breadth.

Continual pre-training: Some deployments update language model weights on a rolling basis as new text data becomes available, adding recent documents while avoiding forgetting of earlier training. In these pipelines, replay is implemented by mixing recently added documents with a sample of older training data at each training step. The mix ratio and the staleness distribution of the replayed documents are key hyperparameters that affect both knowledge freshness and forgetting prevention.

The language modeling context also presents a challenge that is less prominent in classification tasks: positive backward transfer. For LLMs, training on a new domain can sometimes improve performance on related earlier domains, because the new data provides complementary linguistic context that strengthens related representations. Replay methods for LLMs must balance preventing harmful forgetting while not suppressing potentially beneficial interference. This requires evaluation metrics that distinguish harmful forgetting from beneficial interference, rather than relying only on accuracy on past tasks.