Online, Offline, and Continual Learning › Limitations and Impact

The techniques in this chapter are effective but they are not solutions in the sense that an optimizer's convergence proof is a solution. Each one trades one failure mode for another, and the tradeoff has to be managed rather than eliminated.

Offline learning gives you reproducibility at the price of staleness. A frozen model can be audited, versioned, and rolled back without ambiguity. But it cannot notice that the world has changed, and its errors grow silently as the deployment distribution drifts. In domains where the environment is stable, offline training remains the right default, and the machinery of this chapter is unnecessary. In domains where it is not, an offline model is a liability with a known expiration date that nobody has measured.

Online learning gives you adaptivity at the price of stability. Every update changes the policy, every policy change changes the data, and the resulting feedback loop has no natural equilibrium. The three experiments above illustrate the components of the problem: forgetting in the sequential training curves, variance in the streaming trace, and exploitation in the planning gap. None of them is fixed by a single technique. Replay reduces forgetting but slows adaptation. Regularization protects important parameters but can prevent necessary change. Uncertainty penalties avoid exploitation but require calibrated uncertainty, which is itself hard to obtain.

Many continual-learning methods are validated on benchmarks that simplify deployment. A common evaluation protocol presents tasks sequentially with clean boundaries and held-out test sets per task. Real deployments may lack task boundaries, clean test sets, or immediate ground truth for current dynamics. A method that achieves near-zero forgetting on Split CIFAR may still be ineffective when tasks are unlabeled and the distribution shifts continuously rather than in discrete jumps. This benchmark-to-deployment gap remains an active research problem, and we will return to evaluation methodology in Part XI: Evaluation and Understanding.

Memory and privacy are real costs. Replay buffers store raw transitions, which can be large, can contain sensitive data, and can create licensing and compliance obligations that offline training does not. In domains with data-protection requirements, the ability to retain and reuse past experience is constrained by law, not just by memory. Generative replay, which trains a generative model of past data and samples from it instead of storing the data itself, is one response to this constraint, though it introduces its own failure mode: the generative model forgets too, and its samples become a distorted echo of the original data.

Safety claims require care. It is tempting to describe a system with uncertainty gating, shadow deployment, and rollback as "safe." That language overstates what the mechanisms provide. They can limit exposure and shorten the time to detection. They do not guarantee that a neural network will remain accurate, and they cannot detect a shift that leaves no observable trace. The honest claim is that these guardrails can make failure more visible and recoverable, which is a meaningful improvement but not a guarantee.

Sequential adaptation has always faced interference between old and new behavior; replay and continual-learning methods make that tradeoff more explicit and manageable. Replay buffers, Fisher-weighted regularization, and architectural isolation each address it differently, and in combination they can help agents accumulate competence over time instead of cycling through it. That capability supports the lifelong-learning systems that the rest of Part VI and Part VII are working toward.

The limitations are not reasons to abandon the techniques. They are reasons to use them with clear expectations. An online learning system is not a system that has solved the problem of change; it is a system that has made a bet about the rate and form of change and has built mechanisms to manage the consequences when that bet is wrong. The quality of the system depends on how well those mechanisms are matched to the deployment context.