Online, Offline, and Continual Learning › Continual Adaptation Without Unsafe Drift

Suppose you accept that the world changes and decide the model should keep learning online. You set up a replay buffer, a modest learning rate, and a periodic update schedule. What could go wrong?

Quite a lot. Online adaptation in a closed loop is a feedback system, and feedback systems have failure modes that open-loop training does not.

The policy is an optimizer over the model's predictions. Optimizers are adversarial to the objects they optimize. If the model has a region where it over-predicts return, the policy will find that region, regardless of how small it is.

This is the optimizer's curse (Smith and Winkler, 2006): selecting the best of many options according to noisy estimates creates an expected optimism bias in the selected estimate. Larger search sets can intensify that selection pressure, but a realized gap need not grow monotonically with every added candidate. In a world model, "many options" means many candidate action sequences, and modern planning methods may consider thousands.

The consequence for online learning can be severe. Data collection concentrates where the model predicts high return. Training on those transitions can correct the error when real outcomes are recorded and weighted adequately. If updates instead rely on model-generated targets, sparse feedback, or a buffer that underrepresents failures, the loop can reinforce an optimistic error.

It is worth distinguishing two sources of error. The first is ordinary approximation error: the model is not a perfect representation of the environment. The second is selection error: the policy chooses actions based on the model's predictions, and selection error amplifies approximation error in the regions where the model is most optimistic. The optimizer's curse is about selection error, not approximation error. Even a model with small average approximation error can have large selection error if its errors are concentrated in the regions the policy prefers.

We use unsafe drift to describe the slow, cumulative movement of a model into a regime where it is confidently wrong and the policy acts on that confidence. It differs from an ordinary performance regression in three ways.

• It is silent. Open-loop prediction error on the training distribution may look fine, because the agent has stopped visiting the region where the model is wrong.
• It is self-reinforcing. Errors steer the policy, and the policy's data confirms the errors.
• It is non-monotonic in the obvious metrics. A model can improve on average loss while becoming less safe, because the average is dominated by easy regions.

Unsafe drift is not merely hypothetical; it is a plausible failure mode for an agent that adapts its model online without adequate external checks. A concrete example appears in the code below: a planner searching over a wider action range than the model was trained on can select extrapolative actions whose model-predicted return exceeds their true return.

The word "drift" suggests slow motion, but the underlying mechanism can be fast. Once the policy discovers a region of model optimism, it will return there repeatedly, and the data from that region will dominate the training signal. The model becomes more confident about its error, which makes the policy more likely to exploit it, which generates more data from the error region. The process can accelerate rather than stabilize.

There is no single technique that eliminates unsafe drift, but there is a well-established set of guardrails that together make it manageable. They fall into four categories.

Constraint-based. Limit how far the model or policy can move per update. Trust-region methods constrain the KL divergence between successive policies; the same idea applies to models through a bound on the change in predicted transition distributions. A small learning rate with a bounded number of updates per unit of new data is the simplest version of this.

Uncertainty-based. Refuse to act on predictions the model is unsure about. Ensembles can disagree where the data is thin, making disagreement a useful heuristic signal, but they can also agree confidently out of distribution. This connects to Stochasticity and Uncertainty: a well-calibrated epistemic uncertainty estimate should reflect lack of knowledge under relevant shifts. Penalizing high estimated uncertainty can deter some unsupported actions.

Verification-based. Require prespecified evidence before a candidate update takes effect. Shadow deployment runs the new model in parallel with the incumbent and compares task, calibration, and safety metrics on the same live inputs without letting the new model act. A mismatch alone is not a reason to reject it; degradation on the chosen criteria is. Rollback checkpoints improve recoverability when monitoring detects degradation, but cannot guarantee that every harmful change will be detected.

Monitoring-based. Track whether live data lies inside the training distribution. Density models, reconstruction error, ensemble disagreement, and simple feature-space distances all provide signals. An out-of-distribution alarm does not fix anything by itself, but it converts a silent failure into a visible one, which is the precondition for any response.

These categories are complementary rather than alternative. Constraint-based methods can limit the size of a change. Uncertainty-based methods can deter actions in poorly supported regions. Verification-based methods can catch some regressions before promotion, while monitoring can reveal failures after deployment. High-consequence systems often layer all four, and the layers are most effective when designed together.

There is a useful reframing that ties these ideas together. Online adaptation is itself a control problem, with the learning process as the plant. The setpoint is stable performance across the conditions the agent encounters. The control inputs are the learning rate, the replay fraction, the update-to-data ratio, and the decision of whether to accept an update at all. The feedback signal is a monitored metric, ideally one that combines open-loop accuracy with closed-loop task performance.

Seen this way, the online learning rate need not be a hyperparameter tuned only once. A practical heuristic is to let monitored surprise influence it: if recent real-outcome prediction error is high relative to a baseline, cautiously raise the learning rate and lower the old-data replay fraction so fresh data has more weight; if error is low and stable, reverse those changes. This is feedback-driven adaptation. It becomes dual control only when the agent also chooses actions for both their control value and the information they reveal, a connection we will examine in Active Perception, Dual Control, and Exploration.

The control view also clarifies why fixed hyperparameters are often a bad fit for continual learning. A fixed learning rate is like a controller with no feedback. It may be tuned for an average rate of change, but it cannot respond to a sudden shift or a long period of stability. A feedback-driven learning rate can. The cost is that the feedback signal must be chosen carefully, because a noisy or misleading signal can cause the controller to oscillate or to adapt in the wrong direction.

It is worth being explicit about what these guardrails do not provide. They do not give formal guarantees that an arbitrary neural world model will remain accurate. They do not detect shift that produces no observable signature. They do not help if the monitoring metric itself is gamed by the same optimization pressure that drives the policy.

What they aim to provide is a reduction in the expected severity of failure and, more importantly, a reduction in the time to detection. In safety-critical deployment, the second is often worth more than the first, because a system that fails visibly and reverts is generally less dangerous than one that fails silently. We return to these questions in depth in Part XII: Reliable World Models.

There is also a deeper limitation: guardrails are designed against known failure modes. Unsafe drift is a known failure mode, so it can be monitored. But an adaptive system can fail in ways that were not anticipated, and those failures may not trigger any of the monitors. The best defense is a combination of principled design, conservative defaults, and human oversight at the points where the system's behavior changes most.