Replay Methods › Pseudo-Rehearsal
Language AI HandbookAll chapters Replay Methods › Pseudo-RehearsalOne section · Sections in this chapter3 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Experience replay has a practical limitation that becomes acute in real-world deployments: it requires storing raw training data indefinitely. For applications involving sensitive personal information, proprietary data, or data with legal retention constraints, storing past examples may be impermissible. Even without legal constraints, the memory cost grows with the number of tasks and the size of each dataset.
For high-dimensional inputs like raw images, audio waveforms, or long text sequences, even a small buffer of a few thousand examples can require gigabytes of storage. A buffer of 10,000 examples of 224x224 RGB images requires approximately 600 MB of uncompressed storage. Multiply that by the number of models in production and the number of deployment environments, and the infrastructure cost becomes significant.
Pseudo-rehearsal addresses this by sidestepping the storage problem entirely. Instead of saving real past examples, the method generates artificial proxy inputs that, when fed through the network, produce activation patterns similar to those from past tasks. The model then trains on these synthetic examples as if they were real, maintaining old knowledge without retaining any actual historical data.
The Original McCloskey and Cohen Insight
The pseudo-rehearsal concept traces back to Robins (1995), who was inspired by the biological hypothesis that the hippocampus generates replay signals during sleep to consolidate cortical learning. Robins observed that if you train a network on task A and then on task B, it forgets A. But if during task B training you interleave random inputs alongside the real data, and use the network's own outputs on those random inputs as training targets, the network partially preserves its task A behavior.
The key insight is subtle: a randomly generated input has no semantic meaning, but the network's current output encodes a snapshot of what the network knows right now. By training on pairs, you are essentially constraining the network to maintain its current input-output mapping on regions of the input space that don't correspond to the new task. This prevents task B training from completely reorganizing the network's response to all inputs.
Why does this help at all? The intuition is functional covering. A neural network partitions the input space into regions assigned to different outputs. Catastrophic forgetting happens when task B training moves the boundaries of those regions in ways that incorrectly reclassify task A inputs. By showing the model random inputs from various parts of the space and saying "your current output here is correct, maintain it," pseudo-rehearsal anchors the output function at many points simultaneously. Even if the random inputs don't perfectly sample task A's true distribution, they sample the broader input space and prevent the most extreme reorganizations of the model's output function.
def pseudo_rehearsal_step(
model, optimizer, current_batch, input_dim, n_pseudo=64
):
"""
Training step using pseudo-rehearsal with random input generation.
Generates random inputs, uses model's current predictions as pseudo-targets,
and mixes them with the current task batch.
"""
model.train()
X_cur, y_cur = current_batch
# Generate random pseudo-inputs spanning the input space
X_pseudo = torch.randn(n_pseudo, input_dim)
# Capture current model predictions as soft targets (before parameter update)
with torch.no_grad():
pseudo_targets = model(X_pseudo).softmax(dim=-1)
# Loss on current task
logits_cur = model(X_cur)
loss_cur = nn.functional.cross_entropy(logits_cur, y_cur)
# Loss on pseudo-rehearsal: KL divergence from current predictions
logits_pseudo = model(X_pseudo)
log_probs_pseudo = logits_pseudo.log_softmax(dim=-1)
loss_pseudo = nn.functional.kl_div(
log_probs_pseudo, pseudo_targets, reduction="batchmean"
)
total_loss = loss_cur + loss_pseudo
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
return total_loss.item()Using the softmax output as the target rather than the hard argmax preserves more information. Hard targets collapse the model's distribution to a single class; soft targets retain the relative confidence the model assigns to all classes. This is the same insight behind knowledge distillation: soft targets carry richer information about the model's learned representations.
The torch.no_grad() context manager is essential here. The pseudo-targets must be computed from the model's current state before the gradient step, not after. If you compute them after the parameter update, you are training the model to be consistent with its post-update self, which provides no regularization against forgetting. The targets must capture the model's pre-update knowledge snapshot.
Limitations of Pure Pseudo-Rehearsal
Pure pseudo-rehearsal with random inputs has a significant limitation: random inputs from a high-dimensional space rarely fall near the true training distribution. For image tasks, random pixel values produce inputs that look like noise, and the model's behavior on noise inputs may not accurately reflect its behavior on meaningful inputs from past tasks.
Consider a model trained to classify handwritten digits. Its output on a random noise image is essentially arbitrary, constrained only by the architecture's inductive biases. Training on these noise-label pairs preserves the model's behavior on noise, but provides no guarantee about what happens to its behavior on actual digit images. The pseudo-rehearsal signal may be too diffuse to effectively anchor the representations that matter.
To see this more precisely, think about what happens to the decision boundary when the model trains on task B. The gradient signal from task B moves parameters in directions that serve task B. These parameter changes affect the model's output function everywhere in the input space, including in the regions where task A inputs live. If random noise inputs are concentrated in a different part of the input space than task A inputs, they provide no constraint on how the model's behavior changes in task A's region. The pseudo-rehearsal signal and the task A distribution may not overlap at all.
This problem becomes more severe as input dimensionality increases. In a 100-dimensional input space, the volume of the unit ball is vanishingly small relative to the hypercube it is inscribed in. A point drawn uniformly from the hypercube is almost certainly far from the surface of the ball, which is where structured data typically lives. Random noise inputs do not sample the data manifold, and the constraint they provide is irrelevant to maintaining performance on data that does lie on the manifold.
This observation motivates two improvements. First, pseudo-inputs can be drawn from a structured distribution that better approximates the true input distribution. For example, you might use a simple generative model trained on the current task to produce more realistic-looking samples. Second, the rehearsal signal can be applied selectively to network layers or regions of the input space where interference with the new task is most likely. Both improvements require knowing something about the data distribution, which moves pseudo-rehearsal toward generative replay.