Replay Methods › Generative Replay

Generative replay extends pseudo-rehearsal by replacing random noise inputs with samples from a learned generative model. Instead of relying on the hope that random inputs span the relevant input space, generative replay trains an explicit model of each task's data distribution and uses that model to generate high-quality synthetic examples during future training.

The overall system operates as a dual-memory architecture, directly inspired by the hippocampal-neocortical complementary learning systems theory from neuroscience. A generative model (analogous to the hippocampus) stores episodic information about individual tasks and can rapidly reconstruct examples on demand. A task model (analogous to the neocortex) learns the slow, structured knowledge that generalizes across experience. The generative model feeds replayed examples to the task model to guide consolidation.

This approach addresses the data retention problem by compressing each task's distribution into a set of parameters rather than storing raw examples. If the generative model has fewer parameters than the number of raw examples it would take to adequately represent the task, generative replay is strictly more memory-efficient than experience replay. For complex high-dimensional distributions, modern generative models can achieve this compression while maintaining sufficient sample quality for useful replay.

The key word is "sufficient." The samples need not be photorealistic or even individually recognizable. They need to carry enough information about the task distribution that training on them prevents the task model from forgetting. This sets a lower bar than general-purpose generative modeling, and it means that even relatively weak generative models can provide useful replay signals.

The simplest generative replay architecture pairs the task model with a Variational Autoencoder (VAE). The VAE learns a compressed latent representation of each task's data and can generate new examples by decoding random samples from the latent space.

Recall from earlier chapters that a VAE consists of an encoder $q_\phi(z \mid x)$ that maps inputs to a distribution over latent codes, and a decoder $p_\psi(x \mid z)$ that reconstructs inputs from latent codes. The training objective is the Evidence Lower Bound (ELBO):

where:

• $\mathbb{E}_{q_\phi(z|x)}[\log p_\psi(x \mid z)]$: the reconstruction term, measuring how well the decoder can recreate inputs from encoded representations
• $D_{\text{KL}}(q_\phi(z \mid x) \| p(z))$: the regularization term, measuring the KL divergence between the learned posterior and a standard normal prior
• $p(z) = \mathcal{N}(0, I)$: the prior distribution over latent codes, a standard multivariate Gaussian
• $\phi$, $\psi$: the encoder and decoder parameters

During replay, new examples are generated by sampling $z \sim \mathcal{N}(0, I)$ and decoding with $p_\psi(x \mid z)$. These generated samples are then paired with the task model's current predictions as soft labels, following the same distillation logic as pseudo-rehearsal. The key difference from naive pseudo-rehearsal is that the latent space has been explicitly shaped by training to produce meaningful outputs when decoded. Samples from the prior $\mathcal{N}(0, I)$ produce outputs that look like training examples, not random noise.

The VAE approach has an important practical advantage: the latent space is continuous and smooth by design. Small perturbations in the latent code produce smooth changes in the generated output, so the generative model can interpolate between training examples. This makes the generated samples more diverse than simple memorization of training inputs while still remaining realistic. It also means that even if the VAE has not perfectly memorized the training set, the samples it produces are sensible extrapolations rather than garbage.

One limitation of VAE-based replay for image data is that VAE samples tend to be blurry. The reconstruction loss (typically mean squared error) averages over possible outputs rather than selecting the sharpest one, which produces soft, smoothed images. For tasks where fine-grained visual detail is important, this blurriness can reduce the effectiveness of replay. However, for lower-dimensional inputs (tabular data, short embeddings, simple features), VAEs work well and are fast to train.

An alternative to VAEs is Generative Adversarial Networks (GANs), which train a generator and a discriminator in an adversarial loop. The generator learns to produce samples that fool the discriminator into thinking they are real, while the discriminator learns to distinguish real from fake samples. The resulting generator can produce sharper, more realistic samples than VAEs for image data, at the cost of more complex and less stable training.

The GAN training objective is a minimax game between the generator $G$ parameterized by $\theta_G$ and the discriminator $D$ parameterized by $\theta_D$:

where:

• $p_{\text{data}}$: the true data distribution for the current task
• $p(z)$: the prior distribution over generator inputs (typically $\mathcal{N}(0, I)$)
• $D(x) \in [0, 1]$: the discriminator's probability that $x$ is a real example
• $G(z)$: the generator's output given latent code $z$

The discriminator is trained to maximize the objective (correctly labeling real and fake samples), while the generator is trained to minimize it (fooling the discriminator). At equilibrium, the generator distribution matches the true data distribution and the discriminator outputs 0.5 everywhere.

For continual learning with GAN-based replay, the generator is trained alongside the task model. When transitioning to a new task, the current generator produces a large batch of synthetic past-task examples. These synthetic examples are mixed with the new task's real data to form the combined training set for the next phase. After training, a new generator (or updated generator) is trained to capture the distribution of the new task as well.

The main challenge with GAN-based replay is mode collapse: the generator may produce high-quality but low-diversity samples, failing to cover the full distribution of past tasks. If the generator only produces one or two modes of a multi-modal past distribution, replay will fail to prevent forgetting in the underrepresented regions. For example, a generator trained on MNIST digit images might collapse to producing only 3s and 8s. When used for replay, the task model would receive strong signal about those two classes but no signal about the other eight, leading to selective forgetting.

Mode collapse is a long-standing problem in GAN training, and many stabilization techniques have been developed: Wasserstein GAN (WGAN) with gradient penalty, spectral normalization, progressive growing, and feature matching losses. For replay purposes, you typically want diversity more than sharpness, which suggests that training stability and mode coverage should be prioritized over per-sample quality.

A subtle but critical issue with generative replay is that the generative model itself is subject to catastrophic forgetting. When you train the generative model on task 2, it forgets task 1's distribution, and the replay samples it generates are corrupted or entirely from the wrong distribution. The task model receives a degraded replay signal, and forgetting accelerates rather than slows.

This creates a recursive dependency. The task model needs good replay samples to avoid forgetting. Good replay samples require a generative model that remembers past tasks. But the generative model also forgets unless it receives good replay samples. The whole system is susceptible to collapse if the generative model's replay quality degrades too quickly.

The standard solution is to apply replay recursively: the generative model for task $k$ is trained using real data from task $k$ plus synthetic examples generated from the previous generative model for tasks $1, \ldots, k-1$. This creates a generational chain where each new generative model inherits knowledge from the previous one through its generated samples.

This recursive approach works in practice but introduces a compounding error problem. Each generation of the generative model introduces some generation quality loss, and these errors accumulate across tasks. After many tasks, the generated samples for early tasks may be significantly degraded, even if each individual generative model transition introduced only a small quality drop.

The compounding error can be formalized. Suppose the generative model introduces a relative quality loss of $\epsilon$ at each task transition (meaning the average quality of generated task-1 samples decreases by a factor of $1 - \epsilon$ per transition). After $k$ task transitions, the quality of task-1 replay samples is $(1 - \epsilon)^k$, which decays exponentially. For $\epsilon = 0.05$ and $k = 20$ tasks, the quality has dropped to $(0.95)^{20} \approx 0.36$, meaning the samples are only 36% as informative as they were originally.

The severity of this compounding error depends on the fidelity of the generative model. High-quality generators (large VAEs, well-trained GANs) introduce less error per generation, so the error accumulates more slowly. Low-quality generators introduce larger errors that compound rapidly. This creates a practical incentive to invest heavily in generative model quality for long task sequences.

Some methods address the generative model forgetting problem by applying continual learning techniques to the generative model directly. Since the generative model itself is a neural network subject to catastrophic forgetting, you can apply EWC, progressive networks, or other continual learning methods to it just as you would to the task model.

This creates a nested continual learning problem: the outer loop trains the task model using generative replay, and the inner loop maintains the generative model using some other continual learning method. The advantage is that methods like EWC can prevent forgetting in the generative model without requiring the generative model to store past samples recursively. The disadvantage is added complexity: you now have two models to maintain, each with their own continual learning apparatus, and hyperparameter tuning becomes substantially more difficult.

An alternative approach is to use a conditional generative model that takes task identity as an additional input. A single conditional VAE or conditional GAN can learn to generate samples from multiple task distributions by conditioning on a task label. When you want replay samples from task $k$, you provide the task label and sample from the conditional distribution. This sidesteps the forgetting problem in the generative model by treating multi-task generation as a supervised problem: the generator is trained on all tasks simultaneously with task identity as an input feature.

Conditional generative replay requires task labels to be available at generation time, which is typically possible in the offline continual learning setting where task boundaries are clearly defined. In streaming or task-agnostic settings, you must either infer the task label from the input or maintain a separate task identity model.