Replay Methods: Buffer, Pseudo-Rehearsal & Generative Replay

In the previous chapters, we explored the core problem of catastrophic forgetting and how regularization methods like Elastic Weight Consolidation (EWC) address it by penalizing changes to parameters that were important for earlier tasks. Regularization works by constraining the learning process from the inside: it shapes the loss landscape so that gradient descent naturally avoids overwriting critical knowledge. The approach is elegant, but it faces a fundamental scaling challenge. The number of constraints needed to protect $k$ tasks grows with $k$, and the constraints imposed by early tasks can conflict with the plasticity demanded by later ones.

Replay methods take a completely different approach. Rather than constraining how the model learns, they influence what the model learns from. The central idea is simple: if the model forgets old tasks because it never sees data from them again, the obvious fix is to keep showing it that data. Replay methods store or reconstruct examples from past tasks and mix them into the training stream for new tasks, so the model always faces a joint learning problem even when the incoming data stream is sequential. From the optimizer's perspective, each gradient update receives a signal from multiple task distributions simultaneously. There is no forgetting because there is no exclusive focus on any single task.

This intuition connects directly to how biological memory works. The hippocampus is thought to replay neural activity patterns from waking experience during sleep, allowing the neocortex to consolidate memories without catastrophic interference. Computational replay methods are a direct analogue: a memory system stores or generates past experiences, which are periodically replayed to maintain previously learned representations. The theoretical framework of complementary learning systems, developed in cognitive neuroscience to explain why both fast episodic memory (hippocampus) and slow semantic memory (neocortex) are necessary for stable generalization, maps almost exactly onto the architecture of generative replay systems. The hippocampal fast learner captures specific episodes; the neocortical slow learner integrates them into structured knowledge. Replay methods recreate this two-system structure in software.

The replay family contains three main branches. Experience replay stores actual data samples in a buffer and retrieves them during training. Pseudo-rehearsal generates synthetic proxy data to approximate past distributions, avoiding the need to store real examples. Generative replay uses a learned generative model to produce realistic reconstructions of past data on demand. Each branch comes with its own design choices, practical constraints, and failure modes, and together they form the most widely used family of continual learning methods in both research and practice.

Understanding why replay is so effective requires identifying what catastrophic forgetting destroys. When the model trains exclusively on task $T_2$, the gradient signal from that task dominates every parameter update. The loss landscape for task $T_1$ is invisible to the optimizer, so the parameters drift freely in directions that may erase task $T_1$'s knowledge entirely. Replay restores visibility: by including task $T_1$ examples in each batch, the optimizer sees a gradient signal that is a sum over both task distributions. The parameter trajectory is pulled simultaneously toward both tasks' optima, and catastrophic forgetting is replaced by the gentler phenomenon of multi-task interference, where performance on each individual task is somewhat lower than it would be with exclusive training, but no task is catastrophically erased.

Experience replay is the most direct response to catastrophic forgetting: save some examples from each task, and include them in the training batches for all future tasks. When the model trains on task $T_k$, it does not just minimize loss on the current task's data; it also minimizes loss on a small replay sample from every previous task.

The simplicity of this approach belies how much engineering goes into making it work well. The key design decisions revolve around three questions: which examples to store, how many to store, and how to integrate them into training. Each of these decisions has non-trivial consequences for forgetting, plasticity, and computational overhead. A buffer that is too small provides insufficient coverage; a buffer that stores unrepresentative examples may fail to anchor the critical regions of each task's decision boundary; a replay ratio that is too high slows learning on new tasks while one that is too low provides insufficient protection against forgetting.

The replay buffer is a data structure that stores a subset of training examples from past tasks. Its design involves three fundamental questions:

• What to store: Which examples should be retained from each task?
• How much to store: What is the total buffer capacity, and how is it divided across tasks?
• How to use it: When and how should stored examples be mixed into training?

The simplest design answers all three questions with uniform choices: store random examples, divide capacity equally across tasks, and mix replay examples with current task examples in each training batch. This baseline is surprisingly effective and provides a clear point of comparison for more sophisticated approaches.

Formally, let $\mathcal{M}$ denote the replay buffer with total capacity $C$. After observing $k$ tasks, the buffer holds at most $\lfloor C/k \rfloor$ examples from each task. When training on task $T_{k+1}$, each mini-batch contains both examples from the current task and examples randomly sampled from $\mathcal{M}$.

The capacity constraint creates a real tension as the number of tasks grows. With 10 tasks and a buffer of 500 examples, each task gets 50 examples on average. If each task originally had 5,000 training examples, the buffer retains only 1% of each task's information. This compression is not free: the fewer examples the buffer holds for a given task, the less accurately it represents that task's distribution, and the less effective replay becomes at preventing forgetting. The relationship between buffer size and forgetting prevention is roughly logarithmic: doubling the buffer from 50 to 100 examples per task has a larger effect than doubling it from 500 to 1000.

The buffer retains 150 examples total after task 2 has been added. Because each task contributes 150 examples and the buffer evicts the oldest on overflow, the earlier task is gradually replaced. This naive eviction strategy highlights a real problem: with many tasks and a fixed buffer, each individual task is represented by increasingly few examples.

This behavior reflects a fundamental tension between breadth and depth of memory, rather than a software bug. A buffer that tries to remember everything retains so few examples per task that each task's distribution is poorly approximated. A buffer that retains many examples per task quickly runs out of capacity as tasks accumulate. There is no free lunch: you must decide how to allocate a fixed memory budget across a potentially unbounded task sequence.

The naive buffer above overwrites old examples when new ones arrive, which biases the buffer toward recently seen tasks. A better approach is reservoir sampling, which ensures that every example from the entire task sequence has an equal probability of being retained in the buffer at any point.

The algorithm maintains the invariant that after observing $n$ total examples, each one is in the buffer with probability $C/n$, where $C$ is the buffer capacity. When a new example arrives, it replaces a randomly selected existing buffer slot with probability $C/n$:

where:

• $C$: the total buffer capacity
• $n$: the number of examples seen so far (across all tasks)
• $C/n$: the probability that any individual example survives to the current point, which decreases as more examples are seen

The key insight is that this probability decreases over time in a principled way. Early examples are retained with high initial probability but face a growing risk of displacement. Later examples enter with lower probability but are at lower risk of subsequent eviction. The result is a statistically unbiased sample of the entire training history.

To see why the invariant holds, consider what happens when example number $n+1$ arrives. Each of the $n$ previously seen examples currently has probability $C/n$ of being in the buffer (by the invariant). After processing the new example:

• With probability $C/(n+1)$, the new example is stored and replaces one of the $C$ buffer slots uniformly at random.
• Each existing buffer example is evicted with probability $(C/(n+1)) \times (1/C) = 1/(n+1)$.
• Each existing buffer example survives the step with probability $1 - 1/(n+1) = n/(n+1)$.
• The probability that any previously seen example is in the buffer after the update is $(C/n) \times (n/(n+1)) = C/(n+1)$.

This is exactly the target probability for $n+1$ total examples seen. The invariant is maintained inductively. No explicit knowledge of task boundaries is needed, and no task receives preferential treatment regardless of the order examples arrive.

Reservoir sampling achieves a roughly equal distribution across all three tasks despite the buffer capacity being half the size of each individual task's dataset. The slight imbalance is expected statistical noise. No example stream has preferential treatment regardless of the order tasks were encountered.

Reservoir sampling is most appropriate when tasks contribute roughly equal numbers of training examples and when unbiased coverage of all historical data is the goal. In practice, task sizes often differ substantially. If task 1 has 10,000 examples and task 2 has only 100, reservoir sampling will allocate buffer slots proportionally to task size, meaning task 2 receives very few examples. For such imbalanced settings, task-aware allocation strategies that explicitly balance the number of stored examples per task often work better.

Random reservoir sampling provides an unbiased estimate of the task distribution, but it may not be the most effective use of limited buffer capacity. The buffer contains only a small fraction of each task's examples, and which examples are retained can significantly affect learning efficiency. Two tasks with identical class distributions but different example choices can show dramatically different forgetting rates.

The core question is: given a budget of $k$ examples to store from a task with $N$ training examples, which $k$ should you keep? The answer depends on what you believe replay is protecting against. If replay protects the model's overall accuracy on a task, representative examples covering the full class distribution are ideal. If replay protects the model's decision boundary, examples near the boundary are most critical. If replay protects the model's feature representations, examples that activate diverse feature patterns are most valuable.

Several principled selection strategies have been proposed, each encoding different assumptions about what makes an example valuable to replay:

Herding (Prototype-based selection) selects examples whose mean feature vector best approximates the true class mean. After computing embeddings for all examples in a task, the algorithm greedily adds examples that minimize the distance between the selected subset's mean and the full dataset's mean. This ensures the stored examples are representative of each class's central tendency. Herding was introduced in the context of iCaRL, an early and influential class-incremental learning system, where maintaining accurate class means was critical for the nearest-class-mean classifier used at inference time. The greedy algorithm is computationally cheap: it requires one pass to compute the class mean, and then $k$ greedy steps each requiring a single distance computation per candidate.

The greedy herding procedure can be written out explicitly. Let $\mu$ be the true mean embedding of class $c$ across all $N$ training examples. Let $\mathcal{S}$ be the set of selected examples (initially empty) and $P$ be the pool of candidates. At each step:

where $\phi(x)$ is the embedding of example $x$. After selecting $p^*$, it is moved from $P$ to $\mathcal{S}$ and the process repeats. After $k$ steps, $\mathcal{S}$ contains the $k$ examples whose collective mean is closest to $\mu$.

Gradient-based selection keeps examples whose gradients are most aligned with the average gradient of the full task dataset. The idea is that these examples carry the most informative training signal, so replaying them exerts a force similar to training on the full dataset. Computing gradients for every candidate example is expensive, but approximate methods exist. One practical approximation computes a small representative subset of the full-data gradient and selects examples whose individual gradients have the highest cosine similarity to it. Gradient-based selection is conceptually more expensive than herding but can outperform it when class distributions are complex or multi-modal.

Uncertainty-based selection keeps examples the model is currently most uncertain about, often measured by prediction entropy or margin. High-uncertainty examples are close to the decision boundary and most likely to be forgotten or misclassified under parameter drift. The intuition is that the boundary is the most fragile part of the classifier: small parameter changes can flip predictions for examples near the boundary, while interior examples (far from any boundary) resist moderate perturbations. By concentrating the buffer on boundary examples, uncertainty selection maximizes replay's protective effect on classification accuracy.

A practical consideration: uncertainty-based selection measures uncertainty at the time of selection, using the model's current parameters. As the model continues training on new tasks, the decision boundary shifts, and examples that were uncertain during selection may no longer be boundary-proximal. This temporal mismatch limits the effectiveness of uncertainty selection in long task sequences. Some methods address this by periodically re-evaluating and replacing buffer contents.

Each strategy captures a different aspect of the task distribution. Random selection treats all examples equally, producing a buffer that mirrors the overall class proportions but may under-represent the decision boundary. Herding selection clusters around class centroids, giving the replay buffer a strong signal about each class's core characteristics. Uncertainty selection concentrates entirely on the decision boundary region, making it aggressive at preventing boundary drift but at the cost of losing information about the broader class structure.

Which strategy wins depends on the task. For class-incremental learning, where the model must distinguish between all classes seen so far using a single classifier, herding tends to work best because accurate class mean representations are critical. For domain-incremental learning, where the output structure remains fixed but the input distribution shifts, uncertainty-based selection often provides better protection for the classifier boundary. Random selection is a reliable baseline that avoids the overhead of embedding computation or gradient evaluation.

How replay examples are combined with current task data during training affects both the efficiency and stability of continual learning. The two main strategies are interleaved training and alternating batches.

In interleaved training, each mini-batch contains a mix of current task examples and replay samples. If the current task batch size is $B$ and the replay sample size is $R$, then each gradient update uses $B + R$ examples drawn from both distributions simultaneously:

where:

• $\mathcal{B}_{\text{current}}$: the mini-batch of current task examples
• $\mathcal{B}_{\text{replay}}$: the mini-batch sampled from the replay buffer
• $\ell$: the per-example task loss (cross-entropy for classification)
• $\theta$: the model parameters being updated

This uniform weighting treats replay examples the same as current examples. Some variants apply a separate weight $\lambda$ to the replay loss, allowing you to control how aggressively the model is pulled back toward past tasks. Higher $\lambda$ prioritizes stability; lower $\lambda$ allows more plasticity for the new task.

In alternating batches, the training loop alternates between steps on the current task and steps on the replay buffer. This can be useful when the current task and past tasks have very different data modalities or loss scales, and blending them into a single batch would produce unstable gradients. Alternating steps decouple the two optimization signals, which can stabilize training at the cost of slightly slower convergence on each individual task.

The replay_ratio parameter controls the balance between stability and plasticity. A ratio of 0 means pure gradient descent on the current task (maximum plasticity, maximum forgetting). A ratio approaching 1 means the model spends most of its capacity on replayed past tasks (maximum stability, slow learning on new tasks). In practice, ratios between 0.25 and 0.5 work well, though the optimal value depends on task difficulty and the degree of task similarity.

One practical refinement is to use class-balanced sampling from the buffer rather than uniform sampling. If the buffer contains 80 examples from task 1 (class 0 and 1) and 20 from task 2, uniform sampling will over-represent task 1 in each replay batch. Class-balanced sampling ensures each class seen so far contributes an equal number of examples per batch, which is particularly important for class-incremental settings where the classification head must distinguish between a growing set of classes.

A notable variant of experience replay is Dark Experience Replay (DER), which augments the stored examples with the model's logit outputs at the time the examples were added to the buffer. Rather than training on hard labels from the buffer, the model is trained to reproduce its own previous logit distributions on replayed inputs. This is a form of self-distillation: the current model is regularized to maintain consistent output distributions on past examples, not just correct predictions.

The DER loss for a replayed example $(x, z_{\text{stored}})$ where $z_{\text{stored}}$ are the logits stored at insertion time is:

The key insight is that stored logits carry more information than hard labels. A logit vector $[2.3, -1.1, 0.4]$ tells you that the model predicted class 0, how confident it was, and which alternatives it considered plausible. Matching the full logit distribution preserves the final prediction and the internal representation structure of the model at the time of storage. This soft constraint is more forgiving of small parameter changes (the model can drift slightly without penalty) while still preventing large representation shifts.

Experience replay has a practical limitation that becomes acute in real-world deployments: it requires storing raw training data indefinitely. For applications involving sensitive personal information, proprietary data, or data with legal retention constraints, storing past examples may be impermissible. Even without legal constraints, the memory cost grows with the number of tasks and the size of each dataset.

For high-dimensional inputs like raw images, audio waveforms, or long text sequences, even a small buffer of a few thousand examples can require gigabytes of storage. A buffer of 10,000 examples of 224x224 RGB images requires approximately 600 MB of uncompressed storage. Multiply that by the number of models in production and the number of deployment environments, and the infrastructure cost becomes significant.

Pseudo-rehearsal addresses this by sidestepping the storage problem entirely. Instead of saving real past examples, the method generates artificial proxy inputs that, when fed through the network, produce activation patterns similar to those from past tasks. The model then trains on these synthetic examples as if they were real, maintaining old knowledge without retaining any actual historical data.

The pseudo-rehearsal concept traces back to Robins (1995), who was inspired by the biological hypothesis that the hippocampus generates replay signals during sleep to consolidate cortical learning. Robins observed that if you train a network on task A and then on task B, it forgets A. But if during task B training you interleave random inputs alongside the real data, and use the network's own outputs on those random inputs as training targets, the network partially preserves its task A behavior.

The key insight is subtle: a randomly generated input $x_{\text{rand}}$ has no semantic meaning, but the network's current output $f_\theta(x_{\text{rand}})$ encodes a snapshot of what the network knows right now. By training on $(x_{\text{rand}}, f_\theta(x_{\text{rand}}))$ pairs, you are essentially constraining the network to maintain its current input-output mapping on regions of the input space that don't correspond to the new task. This prevents task B training from completely reorganizing the network's response to all inputs.

Why does this help at all? The intuition is functional covering. A neural network partitions the input space into regions assigned to different outputs. Catastrophic forgetting happens when task B training moves the boundaries of those regions in ways that incorrectly reclassify task A inputs. By showing the model random inputs from various parts of the space and saying "your current output here is correct, maintain it," pseudo-rehearsal anchors the output function at many points simultaneously. Even if the random inputs don't perfectly sample task A's true distribution, they sample the broader input space and prevent the most extreme reorganizations of the model's output function.

Using the softmax output as the target rather than the hard argmax preserves more information. Hard targets collapse the model's distribution to a single class; soft targets retain the relative confidence the model assigns to all classes. This is the same insight behind knowledge distillation: soft targets carry richer information about the model's learned representations.

The torch.no_grad() context manager is essential here. The pseudo-targets must be computed from the model's current state before the gradient step, not after. If you compute them after the parameter update, you are training the model to be consistent with its post-update self, which provides no regularization against forgetting. The targets must capture the model's pre-update knowledge snapshot.

Pure pseudo-rehearsal with random inputs has a significant limitation: random inputs from a high-dimensional space rarely fall near the true training distribution. For image tasks, random pixel values produce inputs that look like noise, and the model's behavior on noise inputs may not accurately reflect its behavior on meaningful inputs from past tasks.

Consider a model trained to classify handwritten digits. Its output on a random noise image is essentially arbitrary, constrained only by the architecture's inductive biases. Training on these noise-label pairs preserves the model's behavior on noise, but provides no guarantee about what happens to its behavior on actual digit images. The pseudo-rehearsal signal may be too diffuse to effectively anchor the representations that matter.

To see this more precisely, think about what happens to the decision boundary when the model trains on task B. The gradient signal from task B moves parameters in directions that serve task B. These parameter changes affect the model's output function everywhere in the input space, including in the regions where task A inputs live. If random noise inputs are concentrated in a different part of the input space than task A inputs, they provide no constraint on how the model's behavior changes in task A's region. The pseudo-rehearsal signal and the task A distribution may not overlap at all.

This problem becomes more severe as input dimensionality increases. In a 100-dimensional input space, the volume of the unit ball is vanishingly small relative to the hypercube it is inscribed in. A point drawn uniformly from the hypercube is almost certainly far from the surface of the ball, which is where structured data typically lives. Random noise inputs do not sample the data manifold, and the constraint they provide is irrelevant to maintaining performance on data that does lie on the manifold.

This observation motivates two improvements. First, pseudo-inputs can be drawn from a structured distribution that better approximates the true input distribution. For example, you might use a simple generative model trained on the current task to produce more realistic-looking samples. Second, the rehearsal signal can be applied selectively to network layers or regions of the input space where interference with the new task is most likely. Both improvements require knowing something about the data distribution, which moves pseudo-rehearsal toward generative replay.

Generative replay extends pseudo-rehearsal by replacing random noise inputs with samples from a learned generative model. Instead of relying on the hope that random inputs span the relevant input space, generative replay trains an explicit model of each task's data distribution and uses that model to generate high-quality synthetic examples during future training.

The overall system operates as a dual-memory architecture, directly inspired by the hippocampal-neocortical complementary learning systems theory from neuroscience. A generative model (analogous to the hippocampus) stores episodic information about individual tasks and can rapidly reconstruct examples on demand. A task model (analogous to the neocortex) learns the slow, structured knowledge that generalizes across experience. The generative model feeds replayed examples to the task model to guide consolidation.

This approach addresses the data retention problem by compressing each task's distribution into a set of parameters rather than storing raw examples. If the generative model has fewer parameters than the number of raw examples it would take to adequately represent the task, generative replay is strictly more memory-efficient than experience replay. For complex high-dimensional distributions, modern generative models can achieve this compression while maintaining sufficient sample quality for useful replay.

The key word is "sufficient." The samples need not be photorealistic or even individually recognizable. They need to carry enough information about the task distribution that training on them prevents the task model from forgetting. This sets a lower bar than general-purpose generative modeling, and it means that even relatively weak generative models can provide useful replay signals.

The simplest generative replay architecture pairs the task model with a Variational Autoencoder (VAE). The VAE learns a compressed latent representation of each task's data and can generate new examples by decoding random samples from the latent space.

Recall from earlier chapters that a VAE consists of an encoder $q_\phi(z \mid x)$ that maps inputs to a distribution over latent codes, and a decoder $p_\psi(x \mid z)$ that reconstructs inputs from latent codes. The training objective is the Evidence Lower Bound (ELBO):

where:

• $\mathbb{E}_{q_\phi(z|x)}[\log p_\psi(x \mid z)]$: the reconstruction term, measuring how well the decoder can recreate inputs from encoded representations
• $D_{\text{KL}}(q_\phi(z \mid x) \| p(z))$: the regularization term, measuring the KL divergence between the learned posterior and a standard normal prior
• $p(z) = \mathcal{N}(0, I)$: the prior distribution over latent codes, a standard multivariate Gaussian
• $\phi$, $\psi$: the encoder and decoder parameters

During replay, new examples are generated by sampling $z \sim \mathcal{N}(0, I)$ and decoding with $p_\psi(x \mid z)$. These generated samples are then paired with the task model's current predictions as soft labels, following the same distillation logic as pseudo-rehearsal. The key difference from naive pseudo-rehearsal is that the latent space has been explicitly shaped by training to produce meaningful outputs when decoded. Samples from the prior $\mathcal{N}(0, I)$ produce outputs that look like training examples, not random noise.

The VAE approach has an important practical advantage: the latent space is continuous and smooth by design. Small perturbations in the latent code produce smooth changes in the generated output, so the generative model can interpolate between training examples. This makes the generated samples more diverse than simple memorization of training inputs while still remaining realistic. It also means that even if the VAE has not perfectly memorized the training set, the samples it produces are sensible extrapolations rather than garbage.

One limitation of VAE-based replay for image data is that VAE samples tend to be blurry. The reconstruction loss (typically mean squared error) averages over possible outputs rather than selecting the sharpest one, which produces soft, smoothed images. For tasks where fine-grained visual detail is important, this blurriness can reduce the effectiveness of replay. However, for lower-dimensional inputs (tabular data, short embeddings, simple features), VAEs work well and are fast to train.

An alternative to VAEs is Generative Adversarial Networks (GANs), which train a generator and a discriminator in an adversarial loop. The generator learns to produce samples that fool the discriminator into thinking they are real, while the discriminator learns to distinguish real from fake samples. The resulting generator can produce sharper, more realistic samples than VAEs for image data, at the cost of more complex and less stable training.

The GAN training objective is a minimax game between the generator $G$ parameterized by $\theta_G$ and the discriminator $D$ parameterized by $\theta_D$:

where:

• $p_{\text{data}}$: the true data distribution for the current task
• $p(z)$: the prior distribution over generator inputs (typically $\mathcal{N}(0, I)$)
• $D(x) \in [0, 1]$: the discriminator's probability that $x$ is a real example
• $G(z)$: the generator's output given latent code $z$

The discriminator is trained to maximize the objective (correctly labeling real and fake samples), while the generator is trained to minimize it (fooling the discriminator). At equilibrium, the generator distribution matches the true data distribution and the discriminator outputs 0.5 everywhere.

For continual learning with GAN-based replay, the generator is trained alongside the task model. When transitioning to a new task, the current generator produces a large batch of synthetic past-task examples. These synthetic examples are mixed with the new task's real data to form the combined training set for the next phase. After training, a new generator (or updated generator) is trained to capture the distribution of the new task as well.

The main challenge with GAN-based replay is mode collapse: the generator may produce high-quality but low-diversity samples, failing to cover the full distribution of past tasks. If the generator only produces one or two modes of a multi-modal past distribution, replay will fail to prevent forgetting in the underrepresented regions. For example, a generator trained on MNIST digit images might collapse to producing only 3s and 8s. When used for replay, the task model would receive strong signal about those two classes but no signal about the other eight, leading to selective forgetting.

Mode collapse is a long-standing problem in GAN training, and many stabilization techniques have been developed: Wasserstein GAN (WGAN) with gradient penalty, spectral normalization, progressive growing, and feature matching losses. For replay purposes, you typically want diversity more than sharpness, which suggests that training stability and mode coverage should be prioritized over per-sample quality.

A subtle but critical issue with generative replay is that the generative model itself is subject to catastrophic forgetting. When you train the generative model on task 2, it forgets task 1's distribution, and the replay samples it generates are corrupted or entirely from the wrong distribution. The task model receives a degraded replay signal, and forgetting accelerates rather than slows.

This creates a recursive dependency. The task model needs good replay samples to avoid forgetting. Good replay samples require a generative model that remembers past tasks. But the generative model also forgets unless it receives good replay samples. The whole system is susceptible to collapse if the generative model's replay quality degrades too quickly.

The standard solution is to apply replay recursively: the generative model for task $k$ is trained using real data from task $k$ plus synthetic examples generated from the previous generative model for tasks $1, \ldots, k-1$. This creates a generational chain where each new generative model inherits knowledge from the previous one through its generated samples.

This recursive approach works in practice but introduces a compounding error problem. Each generation of the generative model introduces some generation quality loss, and these errors accumulate across tasks. After many tasks, the generated samples for early tasks may be significantly degraded, even if each individual generative model transition introduced only a small quality drop.

The compounding error can be formalized. Suppose the generative model introduces a relative quality loss of $\epsilon$ at each task transition (meaning the average quality of generated task-1 samples decreases by a factor of $1 - \epsilon$ per transition). After $k$ task transitions, the quality of task-1 replay samples is $(1 - \epsilon)^k$, which decays exponentially. For $\epsilon = 0.05$ and $k = 20$ tasks, the quality has dropped to $(0.95)^{20} \approx 0.36$, meaning the samples are only 36% as informative as they were originally.

The severity of this compounding error depends on the fidelity of the generative model. High-quality generators (large VAEs, well-trained GANs) introduce less error per generation, so the error accumulates more slowly. Low-quality generators introduce larger errors that compound rapidly. This creates a practical incentive to invest heavily in generative model quality for long task sequences.

Some methods address the generative model forgetting problem by applying continual learning techniques to the generative model directly. Since the generative model itself is a neural network subject to catastrophic forgetting, you can apply EWC, progressive networks, or other continual learning methods to it just as you would to the task model.

This creates a nested continual learning problem: the outer loop trains the task model using generative replay, and the inner loop maintains the generative model using some other continual learning method. The advantage is that methods like EWC can prevent forgetting in the generative model without requiring the generative model to store past samples recursively. The disadvantage is added complexity: you now have two models to maintain, each with their own continual learning apparatus, and hyperparameter tuning becomes substantially more difficult.

An alternative approach is to use a conditional generative model that takes task identity as an additional input. A single conditional VAE or conditional GAN can learn to generate samples from multiple task distributions by conditioning on a task label. When you want replay samples from task $k$, you provide the task label and sample from the conditional distribution. This sidesteps the forgetting problem in the generative model by treating multi-task generation as a supervised problem: the generator is trained on all tasks simultaneously with task identity as an input feature.

Conditional generative replay requires task labels to be available at generation time, which is typically possible in the offline continual learning setting where task boundaries are clearly defined. In streaming or task-agnostic settings, you must either infer the task label from the input or maintain a separate task identity model.

Each replay variant occupies a different position in the tradeoff space between memory cost, computational cost, generation fidelity, and forgetting prevention. Understanding when each approach is appropriate requires examining these tradeoffs concretely.

The performance gap between herding-based experience replay and the no-replay baseline illustrates why replay has become the dominant continual learning paradigm in practice. Herding selection retains the most information per buffer slot by choosing representative examples, while the no-replay baseline falls below 20% after five tasks. Generative replay performs similarly to random experience replay, which is encouraging given that it requires no stored data: the cost of compressing task distributions into generative model parameters is roughly equivalent to the cost of storing a small but well-curated random sample.

Pseudo-rehearsal occupies a lower tier in this comparison. Its forgetting protection is consistent across tasks but weaker than methods that retain information from the true distribution. This reflects the fundamental limitation identified earlier: random inputs do not span the true input distribution, so the rehearsal signal does not protect the most important regions of the input-output mapping.

Experience replay grows memory linearly with the number of tasks and buffer size. With $k$ tasks and a buffer of $C$ examples, storage cost is $O(C \cdot d_{\text{input}})$, where $d_{\text{input}}$ is the input dimensionality. For text tasks with long sequences, this can be substantial. For image tasks at typical benchmark resolutions (32x32 to 224x224), even buffers of 10,000 examples are manageable on modern hardware.

Generative replay trades memory for compute. Instead of storing raw examples, it stores generative model parameters. For large datasets, this can be much cheaper: a GAN trained on 100,000 images might have 10M parameters (40 MB at float32), while storing even 5,000 raw images at 224x224 pixels requires similar storage. The key cost is computational: generating replay batches requires forward passes through the generative model at every training step.

Pseudo-rehearsal has negligible memory and compute overhead beyond the base model, but this comes at the cost of replay quality. Random inputs may not effectively protect the decision boundary or class structure of past tasks, especially in high-dimensional spaces.

Replay methods behave differently across the three standard continual learning settings, and understanding these differences is important for applying them correctly.

In the task-incremental setting, the model has access to the task identity at inference time. A separate classification head is typically maintained for each task, and the model simply routes each test example to the correct head based on the provided task label. In this setting, forgetting is almost entirely a representation learning problem: the shared feature extractor must maintain useful representations for all tasks, but the task-specific heads protect against direct output interference. Replay helps maintain the quality of shared representations, and relatively small buffers are sufficient.

In the class-incremental setting, the model must classify among all classes seen so far without knowing which task the test example came from. This is substantially harder. The classification head must grow as new classes are added, and it must simultaneously distinguish between classes from all tasks. Here, replay is critical for representations and for the classifier boundary. Without replay of past task examples, the growing classification head will dominate the loss during new task training, causing the features and decision boundaries for past classes to degrade. Herding-based replay was specifically designed for this setting, where maintaining accurate class mean representations is key to the nearest-class-mean classification approach used in iCaRL.

In the domain-incremental setting, the output structure remains fixed but the input distribution shifts across tasks. For example, a sentiment classifier first trained on movie reviews and then adapted to product reviews must maintain performance on both domains with the same output classes. Replay in this setting focuses on anchoring the input-output mapping in the original domain's distribution, making uncertainty-based selection (which prioritizes boundary examples) particularly effective.

The standard Split-CIFAR and Permuted-MNIST benchmarks that appear throughout the continual learning literature each test a different variant of these settings. Split-CIFAR, where CIFAR-10 is divided into 5 two-class tasks presented sequentially, most closely approximates the class-incremental scenario with the same output space. Permuted-MNIST, where each task applies a different fixed permutation to the input pixels, approximates the domain-incremental scenario.

The replay intuitions developed for smaller classification models become more complex when applied to large language models. LLMs present several unique challenges that require adaptations to the standard replay framework.

Dataset scale: Pre-training datasets for LLMs contain trillions of tokens. Storing even a tiny fraction of this data for replay is prohibitively expensive. In practice, replay for LLMs uses document-level examples rather than individual training instances, and buffer capacity is measured in millions rather than thousands of examples. The per-example cost is also higher: a single long-context document may contain thousands of tokens, making each replay example significantly more memory-intensive than a single image or feature vector.

Implicit replay through data mixing: Many LLM training pipelines include replay implicitly. When fine-tuning a language model on a new domain, practitioners often mix in a fraction of pre-training data to maintain general language understanding. This is experience replay at scale, though it is typically not framed that way in the literature. The mixing ratio, the pre-training data selection strategy, and the curriculum order are all forms of replay hyperparameters, even if they are not explicitly identified as such.

Instruction tuning and RLHF: When fine-tuning with instruction data or reinforcement learning from human feedback, replay of base pre-training examples or earlier fine-tuning data is commonly used to prevent the model from losing its general capabilities. The replay fraction is a critical hyperparameter: too little and the model forgets; too much and it fails to acquire the new behavior. Studies on instruction-tuned models have found that including even 5-10% general pre-training data in the fine-tuning mix substantially preserves general capability while allowing the model to acquire task-specific behavior.

Catastrophic forgetting in fine-tuning: The forgetting problem becomes especially visible when fine-tuning on narrow domains. A model fine-tuned extensively on medical texts will lose its ability to discuss other topics, answer general knowledge questions, and maintain coherent multi-turn conversations. Mixing a small amount of general web data into the fine-tuning corpus is a practical replay strategy that significantly mitigates this degradation at low cost. The selection of which web documents to include can itself be optimized: documents that are semantically distant from the fine-tuning domain provide the most complementary signal for maintaining breadth.

Continual pre-training: Some deployments update language model weights on a rolling basis as new text data becomes available, adding recent documents while avoiding forgetting of earlier training. In these pipelines, replay is implemented by mixing recently added documents with a sample of older training data at each training step. The mix ratio and the staleness distribution of the replayed documents are key hyperparameters that affect both knowledge freshness and forgetting prevention.

The language modeling context also presents a challenge that is less prominent in classification tasks: positive backward transfer. For LLMs, training on a new domain can sometimes improve performance on related earlier domains, because the new data provides complementary linguistic context that strengthens related representations. Replay methods for LLMs must balance preventing harmful forgetting while not suppressing potentially beneficial interference. This requires evaluation metrics that distinguish harmful forgetting from beneficial interference, rather than relying only on accuracy on past tasks.

Replay methods are not a complete solution to catastrophic forgetting. Several fundamental limitations constrain their effectiveness in practice, and understanding these constraints helps you make better design decisions and set realistic expectations.

Privacy and data retention: Experience replay requires storing raw training data, which may include private user information, proprietary documents, or personally identifiable data. Regulations like GDPR's right to erasure explicitly prohibit storing data past its approved retention period, making experience replay legally impermissible in many deployment contexts. This is not a theoretical concern: several high-profile machine learning deployments have faced regulatory challenges related to training data retention. Generative replay sidesteps this by storing a generative model rather than raw data, but the generative model may itself memorize and leak sensitive training information. This has been demonstrated empirically: large neural networks can be probed to recover training examples, and generative models are particularly susceptible because they are explicitly trained to reproduce their training distribution. Even generative replay is not a complete privacy solution.

Forgetting is not fully eliminated: Even with replay, forgetting is slowed but not stopped. With a fixed buffer capacity, each new task dilutes the representation of all previous tasks. After 100 tasks with a buffer of 1,000 examples, each task receives only 10 replay examples on average, which may be insufficient to prevent significant forgetting. The fundamental constraint is that a model of fixed capacity cannot learn an unlimited number of tasks without some loss of precision. Replay can push the Pareto frontier of the forgetting-plasticity tradeoff outward, but it cannot eliminate the tradeoff.

This limitation motivates the concept of forward transfer: rather than trying to prevent all forgetting, you might accept some degradation on old tasks while ensuring that new task learning benefits from previous experience. A model that forgets 10% on old tasks but learns new tasks 30% faster has a better overall information efficiency than one that forgets nothing but learns each new task from scratch. Replay methods that focus exclusively on preventing forgetting may miss this opportunity.

Task boundary assumption: Most replay methods assume that task boundaries are known: you know when the training distribution shifts and can trigger the replay logic accordingly. In online or streaming settings where task boundaries are blurred or unknown, identifying when and how to update the buffer and generative model is non-trivial. You need either a change detection mechanism to identify distributional shifts, or a continuous updating strategy that gradually incorporates new data without relying on explicit task boundaries. The latter is the subject of ongoing research in task-agnostic continual learning.

Distribution shift in generative replay: The quality of generative replay degrades as the generative model itself accumulates forgetting across many task transitions. This generational degradation is a fundamental limitation: the replay signal becomes noisier over time, potentially accelerating rather than preventing forgetting in long task sequences. The compounding error grows exponentially with the number of tasks, which means generative replay is most reliable for short task sequences (5-20 tasks) and becomes increasingly unreliable for longer sequences unless very high-quality generative models are used.

Computational overhead: Interleaving replay examples adds computational cost proportional to the replay ratio. For tasks where forward and backward passes are expensive (large LLMs, high-resolution image models), even a 25% replay overhead adds significant wall-clock time to training. For a model whose training step costs 1 second, a replay ratio of 0.5 roughly doubles training time. This cost must be weighed against the forgetting prevented. In settings where forgetting is tolerable or where the new task is so different from past tasks that replay provides little benefit, skipping replay entirely may be the correct engineering decision.

Buffer management complexity: Maintaining a replay buffer in production requires careful engineering. The buffer must be serialized and loaded with the model, updated atomically when new tasks arrive, and queried efficiently during training. For distributed training across many accelerators, the buffer must be replicated or sharded consistently. These engineering requirements are often underestimated in research settings but become significant obstacles in production deployments.

Despite these limitations, replay methods remain among the most effective and widely deployed continual learning techniques. Experience replay in particular has repeatedly demonstrated competitive performance on standard benchmarks, often outperforming both regularization methods and architecture methods for modest buffer sizes. Its premise is straightforward: continued practice helps preserve a learned behavior.

Replay methods address catastrophic forgetting by making the training distribution over all tasks visible to the model even as new tasks arrive sequentially. The core idea: if forgetting happens because the model stops seeing old data, the fix is to keep showing it old data, either directly through stored examples or indirectly through synthetic reconstructions.

The key ideas from this chapter are:

• Experience replay stores raw past examples in a buffer and mixes them into training batches for new tasks. Reservoir sampling ensures unbiased coverage of all past tasks. Selection strategies (herding, gradient-based, uncertainty-based) optimize which examples to store for maximum protective effect per slot.

• Dark Experience Replay extends experience replay by storing the model's logit outputs alongside each saved example. Training the model to reproduce its previous logit distributions, rather than just correct labels, preserves richer representation structure and more effectively prevents forgetting with the same buffer budget.

• Pseudo-rehearsal generates synthetic proxy inputs and uses the model's own outputs as training targets, preserving old input-output mappings without storing any real data. Random inputs are cheap and simple but fail to span the true input distribution in high-dimensional spaces, limiting their effectiveness.

• Generative replay trains a generative model (VAE or GAN) on each task and uses it to produce realistic synthetic examples during future training. The dual-memory architecture parallels hippocampal-neocortical complementary learning. However, the generative model itself is subject to forgetting and introduces compounding generation quality loss across many task transitions.

• Replay selection matters significantly for small buffers. Herding selection outperforms random sampling in class-incremental settings, while uncertainty-based selection is more effective in domain-incremental settings. The optimal strategy depends on the task structure and the continual learning scenario.

• LLM replay takes the form of data mixing: including a fraction of pretraining or earlier fine-tuning data in subsequent fine-tuning runs. This is experience replay at scale, and the mixing fraction is one of the most important hyperparameters for preventing capability degradation in fine-tuning pipelines.

The choice between experience replay, pseudo-rehearsal, and generative replay depends on your constraints. If you can store data and have no privacy restrictions, experience replay with careful selection is the most reliable option. If data retention is impossible, generative replay offers a principled alternative at the cost of generative model complexity and the risk of compounding generation errors. If neither memory nor compute budget allows for sophisticated replay, pseudo-rehearsal provides a lightweight fallback with partial forgetting protection. The right answer is almost always some form of replay. The question is which form your deployment constraints permit.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about replay methods in continual learning.