Online, Offline, and Continual Learning

A warehouse robot spends its first month learning how boxes slide when pushed. The model is good. Then the facility installs a new conveyor belt, and the same pushes produce different outcomes. The robot's world model is now wrong, and it has no way of knowing that unless it keeps learning. If it keeps learning, it risks overwriting the month of experience it accumulated. And if it adapts too eagerly to the new belt, it may become worse at everything else.

This chapter is about that tension. Every world model we have built in this handbook was trained by minimizing some loss on some dataset, as we saw in Predictive and Generative Objectives and Data Curation and Synthetic Data. But real agents do not stop collecting data when training ends. They keep acting, keep observing, and keep receiving new transitions that may or may not resemble the ones they were trained on. The question is how to keep the model useful over time.

That question changes what a world model is. A model trained once on a fixed dataset is a snapshot of the world as it was during data collection. A model that keeps learning is a process that must decide, at every update, how much to trust the past and how much to trust the present. The design choices that govern that decision determine whether the agent accumulates competence, remains frozen in an obsolete regime, or chases its own errors into regions where it is confidently wrong.

We will work through three regimes and the problems that separate them.

• Offline learning fits a model once to a fixed dataset. It is reproducible, easy to evaluate, and easy to audit.
• Online learning interleaves updates with data collection. It is adaptive, but every update changes the data the agent will collect next.
• Continual learning faces a sequence of tasks, environments, or regimes and must perform well on all of them, not only the most recent one.

Each regime brings its own failure mode. Offline models go stale. Online models drift. Continual models forget. These are not three separate problems to be patched independently; they are three faces of a single design decision about how much weight the model should give to old data relative to new data. That decision is the subject of this chapter.

The regimes also differ in what information is available at training time. In the offline regime, the dataset is fixed before optimization begins, so every update can be evaluated against a stable objective. In the online regime, the objective itself is moving because the data distribution is generated by a policy that the model is helping to change. In the continual regime, there may be no single objective at all, only a sequence of objectives that may conflict. Each transition between regimes removes an assumption that made the previous regime easier to analyze.

We will also keep a distinction in view that runs through the whole book. A world model can be evaluated in two very different ways. Open-loop accuracy asks whether predicted futures match observed futures. Closed-loop usefulness asks whether an agent that plans with the model achieves its goals. A model can improve on the first metric while degrading on the second, because the policy is an optimizer and optimizers find the model's mistakes. Any discussion of online adaptation has to hold both metrics at once.

That distinction is worth making concrete early. Open-loop accuracy is measured by holding the model fixed, feeding it a sequence of inputs from a held-out dataset, and comparing its predictions to the recorded targets. Closed-loop usefulness is measured by placing the model inside an agent, letting the agent choose actions, and evaluating the outcomes in the real environment. The first metric treats the model as a predictor. The second treats it as a component whose errors alter the distribution of future inputs. A model can be excellent as a predictor on the data it has seen and catastrophic as a component, because the agent will deliberately steer toward the parts of the input space where the model's predictions are most favorable, not necessarily most accurate.

The definition above contains a subtle phrase: "presented sequentially." A continual learning problem is more than a collection of datasets because the order matters. If the agent sees task A and then task B, the update for task B can erase task A. If it sees them interleaved, the interference may be milder. If it sees task A, then task B, then task A again, it may recover some performance on A, but only by damaging B. Continual learning is therefore a study of trajectories through parameter space, not only final loss values on individual tasks.

The distinction between batch and streaming updates is about how data reaches the optimizer, not about how the optimizer works.

In the batch (or offline) setting you have a dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ collected beforehand. You choose parameters by minimizing the empirical risk

$$\hat{\theta} = \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} \ell\big(f_\theta(x_i),\, y_i\big).$$

where:

• $\hat{\theta}$: the parameter estimate that minimizes the average loss over the dataset
• $\theta$: the parameters being optimized
• $\mathcal{D}$: the dataset of input-target pairs, $\{(x_i, y_i)\}_{i=1}^N$
• $N$: the number of examples in the dataset $\mathcal{D}$
• $\ell(\cdot, \cdot)$: the per-example loss function, such as squared error
• $f_\theta(x_i)$: the model's prediction for input $x_i$
• $y_i$: the target value paired with $x_i$
• $x_i$: the $i$-th input example

For a world model, $x_i$ is a transition input such as $(s_t, a_t)$, $y_i$ is the target such as $s_{t+1}$, and $f_\theta$ is the learned dynamics. This is the setting of Part V and Part VI so far. It has several properties worth naming explicitly:

• Reproducibility. The result can be reproduced when the dataset, architecture, optimization procedure, software stack, hardware, and random seeds are controlled.
• Multiple passes. You can shuffle and revisit every example as many times as you like.
• Auditability. You can inspect exactly what the model was trained on.
• Staleness. Once training stops, the model has no mechanism to notice that the world changed.

These properties are tightly connected. Reproducibility comes from the fact that the dataset is fixed, so the same optimization procedure applied to the same data yields the same result up to random seeds and hardware nondeterminism. Multiple passes are possible because the dataset is stored in full and can be revisited indefinitely; this is what allows offline training to extract many gradient steps per example. Auditability follows from the same storage: the training distribution is not a moving target but a concrete artifact that can be inspected, versioned, and checked for biases or gaps. Staleness is the price. A model that was trained on a fixed dataset has no internal signal that the world has moved on. Its errors on the new world are not necessarily large, but they are unmonitored.

In the streaming (or online) setting data arrives as a sequence $(x_1, y_1), (x_2, y_2), \ldots$ and you update after each arrival:

$$\theta_{t+1} = \theta_t - \eta_t \,\nabla_\theta\, \ell\big(f_{\theta_t}(x_t),\, y_t\big).$$

where:

• $\theta_{t+1}$: the parameters after processing the $t$-th example
• $\theta_t$: the parameters before the update
• $\eta_t$: the learning rate at step $t$
• $t$: the current step index in the stream
• $x_t, y_t$: the $t$-th input and target pair arriving from the stream
• $\ell(\cdot, \cdot)$: the per-example loss function
• $\nabla_\theta \ell(f_{\theta_t}(x_t), y_t)$: the gradient of the loss on the current example with respect to the parameters
• $f_{\theta_t}(x_t)$: the model's prediction for input $x_t$ using the current parameters
• $\theta$: the model parameters

This is stochastic approximation. Under the usual regularity assumptions on the objective, gradients, and noise, a standard route to convergence uses the classical Robbins--Monro step-size conditions. They require that the learning rates accumulate enough total step size to keep moving while the squared learning rates accumulate little enough total noise for it to average out:

$$\begin{aligned} \sum_{t=1}^{\infty} \eta_t &= \infty && \text{(step sizes must not shrink too fast)} \\ \sum_{t=1}^{\infty} \eta_t^2 &< \infty && \text{(squared step sizes must shrink fast enough)} \end{aligned}$$

where:

• $\eta_t$: the learning rate applied at step $t$
• $\sum_{t=1}^{\infty} \eta_t$: the total accumulated step size, which must diverge so the estimator can travel any distance
• $\sum_{t=1}^{\infty} \eta_t^2$: the total accumulated squared step size, which must converge so gradient noise averages out

Here $\eta_t$ is the learning rate applied at step $t$. Such schedules are called square-summable but not summable: the first condition keeps the estimator moving in expectation, while the second keeps its variance from accumulating.

The first condition says the learning rate must not shrink so fast that the model freezes before it arrives. The second says it must shrink fast enough that accumulated gradient noise averages out. A schedule like $\eta_t = 1/t$ satisfies both. The schedule $1/\sqrt{t}$ and a positive constant both violate square summability; the constant still satisfies the divergent-sum condition. That persistent step size is useful in non-stationary settings because it keeps the estimator responsive, at the cost of a nonzero noise floor.

The schedule shape determines how the estimator behaves over time. The figure below compares three common choices on logarithmic axes, where the difference between square-summable and non-summable decay is easiest to read.

The Robbins and Monro conditions are a useful reference point even when they do not literally apply. They describe the behavior of an estimator that is trying to converge to a fixed target in a stationary world. In a world-modeling loop, the target is not fixed: the true dynamics may change, and even if they do not, the distribution of states and actions that the model sees depends on the policy, which depends on the model. The classical conditions therefore tell us what convergence would require if the world stood still. They also tell us why practical systems use constant or slowly decaying learning rates: the agent is not trying to converge to a fixed answer, because the answer keeps moving.

A useful mental model is to separate two questions. The first is whether the optimizer can reach a good solution if one exists. The second is whether the optimizer can track a solution that changes over time. Batch learning is optimized for the first question. Streaming learning with a constant learning rate is optimized for the second. Continual learning must answer both, often without knowing which regime it is currently in.

Batch and streaming are endpoints of a spectrum. Full-batch gradient descent uses all $N$ examples per step. Mini-batch SGD uses $B$ examples per step. Pure online SGD uses one. Everything between is a design choice about how much of the historical distribution to include in each update. A replay buffer, which we turn to in the next section, is the device that lets you move along this spectrum even when data arrives one sample at a time.

The spectrum covers both batch size at a single step and how long data is retained. A method could update on a mini-batch of recent data and still be highly online if that mini-batch is discarded immediately. Another method could update on a single example drawn from a large historical buffer and behave much more like batch training. The relevant quantity is the effective age distribution of the data contributing to each update.

There is a precise sense in which the learning rate controls how much history the estimator remembers. Consider the simplest case: a scalar linear model $y_t = \theta^\star x_t + \epsilon_t$ with $\epsilon_t$ zero-mean and independent of $x_t$. The SGD update is

$$\theta_{t+1} = \theta_t - \eta\,(\theta_t x_t - y_t)\,x_t.$$

where:

• $\theta_{t+1}$: the updated parameter estimate
• $\theta_t$: the current parameter estimate
• $\eta$: the constant learning rate
• $x_t$: the scalar input at step $t$
• $y_t$: the observed target at step $t$
• $\theta_t x_t - y_t$: the prediction error, which drives the update

Here $\theta_t$ is the current parameter estimate, $\eta$ is the constant learning rate, $x_t$ is the scalar input at step $t$, $y_t$ is the observed target, and $\theta_t x_t - y_t$ is the prediction error that drives the update.

Writing the error as $e_t = \theta_t - \theta^\star$ and substituting $y_t$ gives

$$e_{t+1} = \big(1 - \eta x_t^2\big)\, e_t + \eta x_t \epsilon_t.$$

where:

• $e_{t+1}$: the parameter error after the update, $\theta_{t+1} - \theta^\star$
• $e_t$: the parameter error before the update, $\theta_t - \theta^\star$
• $\eta$: the learning rate
• $x_t$: the scalar input at step $t$
• $\epsilon_t$: the zero-mean observation noise at step $t$
• $\theta_t$: the parameter estimate before the update
• $\theta_{t+1}$: the parameter estimate after the update
• $\theta^\star$: the true parameter value

The first term $(1 - \eta x_t^2) e_t$ shrinks the existing error, while the second term $\eta x_t \epsilon_t$ injects noise. Take expectations over the noise and inputs, assuming the current sample is independent of $e_t$, $\mathbb{E}[x_t\epsilon_t]=0$, and $\mathbb{E}[x_t^2]=\sigma_x^2$:

$$\mathbb{E}[e_{t+1}] = \big(1 - \eta \sigma_x^2\big)\,\mathbb{E}[e_t].$$

where:

• $\mathbb{E}[e_t]$: the expected parameter error at step $t$
• $\eta$: the learning rate
• $\sigma_x^2$: the variance of the inputs $x_t$
• $1 - \eta \sigma_x^2$: the geometric decay rate of the bias
• $\mathbb{E}[e_{t+1}]$: the expected parameter error at step $t+1$

The bias decays geometrically at rate $1 - \eta\sigma_x^2$. A large learning rate forgets old errors quickly. A small one holds onto them. We will return to this in the worked example.

This recursion is the simplest possible model of memory in gradient descent. The current error is a weighted combination of the previous error and a noise term. The weight on the previous error is approximately one minus the learning rate times the input variance. If that weight is close to one, the estimator remembers its old error for a long time. If it is close to zero, the estimator discards its old error almost immediately. In a stationary world, a small weight is good, because the old error was itself caused by noise or by initialization and should be forgotten. In a non-stationary world, the old error may contain information about dynamics that are still partially valid, and forgetting it completely may be wasteful.

There is a second sense in which the learning rate controls memory. The stationary variance of the estimator is proportional to the learning rate. A large learning rate makes the estimator responsive but noisy. A small learning rate makes it precise but slow. The memory knob and the noise knob are the same knob, turned in opposite directions. This is the fundamental tradeoff that reappears in every method in this chapter.

For linear models there is an exact streaming estimator that avoids gradient descent entirely. Recursive least squares (RLS) maintains a parameter estimate $\theta_t$ and a covariance matrix $P_t$:

$$\begin{aligned} P_t &= P_{t-1} - \frac{P_{t-1} \phi_t \phi_t^\top P_{t-1}}{1 + \phi_t^\top P_{t-1} \phi_t}, \\ \theta_t &= \theta_{t-1} + P_t \phi_t \big(y_t - \phi_t^\top \theta_{t-1}\big). \end{aligned}$$

where:

• $P_t$: the inverse information matrix after step $t$; under a matching linear-Gaussian model with normalized observation noise, it is proportional to the posterior covariance of $\theta$
• $P_{t-1}$: the covariance before the update
• $\phi_t$: the input (feature) vector at step $t$
• $\theta_t$: the parameter estimate after step $t$
• $\theta_{t-1}$: the parameter estimate before the update
• $y_t$: the target at step $t$
• $y_t - \phi_t^\top \theta_{t-1}$: the prediction residual, which drives the parameter correction
• $\theta$: the parameter vector being estimated

If you have read Bayesian Filtering and Belief States, this should look familiar: RLS is a Kalman filter for a constant parameter with no process noise. Under a matching linear-Gaussian model, $P_t$ is proportional to the posterior covariance of $\theta$, and it shrinks as evidence accumulates. With persistent excitation in every parameter direction, $P_t$ approaches zero and the gain becomes arbitrarily small. Plain RLS then adapts increasingly slowly to a change; directions that are not sufficiently excited need not collapse.

The intuition is that RLS is not merely averaging recent samples. It is maintaining a full second-order description of what it knows. When data is plentiful and the model is confident, the covariance becomes small, so the update gain becomes small. When data is scarce or the input directions are poorly explored, the covariance remains large, so the estimator is willing to move. This is exactly the behavior a learner should have in a stationary world. It is exactly the wrong behavior in a world that changes, because the estimator becomes most confident precisely when it has seen the most data, which may be the data from before the change.

The fix is to add process noise, which produces a forgetting factor $\lambda$:

$$P_t = \frac{1}{\lambda}\left(P_{t-1} - \frac{P_{t-1} \phi_t \phi_t^\top P_{t-1}}{\lambda + \phi_t^\top P_{t-1} \phi_t}\right).$$

where:

• $\lambda$: the forgetting factor, with $0 < \lambda \le 1$; smaller $\lambda$ down-weights older data more aggressively and inflates the covariance to preserve plasticity
• $P_t, P_{t-1}$: the covariance after and before the update
• $\phi_t$: the input vector at step $t$

With $\lambda < 1$, older observations are exponentially down-weighted with time constant $1/(1-\lambda)$. This is the cleanest formal statement of the memory-versus-plasticity tradeoff: the forgetting factor is a literal discount on history. A value of $\lambda = 0.99$ means the effective window is roughly 100 samples; $\lambda = 0.999$ means roughly 1000.

The forgetting factor has a statistical interpretation: it inflates the covariance carried from the previous step, expressing a belief that old information is becoming less relevant. In a Kalman interpretation this resembles process noise, although exponential forgetting generally implies a state-dependent covariance inflation rather than a fixed random-walk covariance. In a world model, the drifting quantity might be a physical coefficient, a reward function, or an entire latent dynamics model.

A quantity that matters in every model-based agent is the update-to-data ratio, sometimes called the replay ratio: how many gradient steps the model takes per environment step collected. In offline training the ratio is enormous (many epochs over a fixed dataset). In a purely streaming loop it is one. In practice, agents such as Dreamer and MBPO sit somewhere in between, and the choice has real consequences. A high ratio extracts more information per real interaction, which is the entire appeal of learning a model in the first place. But a high ratio also means the model is repeatedly optimized against a fixed, possibly stale dataset, so it overfits to whatever the last batch of data happened to contain. Modern practice typically sets an explicit update budget and monitors both open-loop and closed-loop metrics against it.

The update-to-data ratio is one of the few quantities in model-based learning that directly connects computational budget to statistical behavior. If the ratio is one, the model sees each transition exactly once and then moves on. If the ratio is large, the model may see the same transition many times, which can be useful when real interaction is expensive but dangerous when the transition is no longer representative. A high ratio also changes the effective learning rate schedule: many small updates on the same data can approximate a larger update, and many updates on stale data can drive the model to memorize idiosyncrasies of the buffer.

The streaming update above assumes the data are drawn from a stationary distribution and that consecutive samples are independent. Neither assumption holds in a world-modeling loop.

• Stationarity fails whenever the environment changes, the agent's policy changes, or the agent's own learning changes what it does.
• Independence fails because consecutive transitions in a trajectory are highly correlated. A robot that has just turned left is likely to turn left again.

Correlated samples make the gradient an average over a stretch of trajectory rather than over the distribution of interest, which biases updates and inflates effective variance. This is one reason practical systems shuffle, buffer, and batch rather than truly updating one sample at a time. It is also why the word "online" in modern deep RL usually means "interleaved with data collection," not "one gradient step per sample with no memory."

There is a further assumption that is often invisible: the loss being minimized is assumed to be the right objective for the deployment distribution. In a stationary supervised problem, minimizing average loss on the training distribution is a reasonable proxy for minimizing loss on the test distribution. In a world model, the training distribution is generated by the agent's own behavior, and the test distribution is whatever distribution the agent will encounter in the future. Those two distributions can diverge dramatically. Minimizing loss on yesterday's data may not reduce loss on tomorrow's data if the policy has moved on.

Distribution shift is the situation where the data the model sees at deployment differs from the data it was trained on. It is the reason an offline model goes stale and the reason an online model can chase its own tail.

The classical taxonomy distinguishes three kinds of shift, and it is worth mapping each onto world models explicitly.

• Covariate shift: the distribution of inputs $p(x)$ changes, but the conditional $p(y \mid x)$ does not. In a world model this means the agent visits new states and new actions, but the underlying physics is unchanged. This happens constantly, because policies explore.
• Concept shift: the conditional $p(y \mid x)$ changes while inputs stay in the same range. In a world model this is a real change in dynamics: the conveyor belt speeds up, the surface becomes slippery, gravity changes in simulation. This is what we usually mean by the world changing.
• Prior-probability shift: the label prior $p(y)$ changes while the class-conditional input distribution $p(x \mid y)$ remains fixed. In world-model deployments, analogous changes in goal or task frequencies can alter what the planner encounters even when the transition dynamics are unchanged.

The distinction matters because the remedies differ. Covariate shift can often be handled by collecting more data in the new input regions, because the underlying mapping is unchanged. Concept shift cannot be fixed by coverage alone; the model must update its parameters to reflect a new mapping. Prior shift may not require any change to the dynamics model at all if the dynamics are unchanged, but it can still matter for planning if the objective or the distribution of goals has changed.

A supervised model that experiences covariate shift can make worse predictions on new inputs. A world model inside an agent can also induce further shift. The policy queries the model, the model returns predictions, the policy acts on those predictions, and the resulting transitions become the next training data. Model error can therefore alter the state distribution, which then changes what the model learns next. This feedback is what makes distribution shift in world models qualitatively different from distribution shift in ordinary supervised learning.

There are two consequences worth separating.

First, extrapolation is not rare, it is systematic. The policy is chosen to maximize expected return under the model. By construction, it prefers states and actions where the model predicts high return. If the model is optimistic anywhere, the policy will find that region. This is the optimizer's curse, and we will demonstrate it concretely in the code.

Second, the training distribution can narrow over time. If the agent only trains on data from the current policy, it may gradually stop seeing states that earlier policies visited. The model can become confident about a shrinking region and increasingly wrong outside it. This creates the conditions for a self-confirming loop: model errors steer the policy, and the resulting data can entrench those errors.

A useful way to see the difference from supervised learning is to imagine a classifier that is used to decide which emails reach each inbox. If the classifier makes mistakes, recipients see worse emails, but the distribution of incoming emails does not change because of the classifier's decisions. In a world model, the policy is like a classifier that decides which states to visit, and those states become the training data for the next round. The classifier is not a passive observer of its input distribution; it is an active participant in generating that distribution.

The most widely used defense is experience replay. Instead of updating only on the newest transition, the agent stores transitions in a buffer $\mathcal{B}$ and samples mini-batches from it:

$$\theta \leftarrow \theta - \eta\, \nabla_\theta \frac{1}{|\mathcal{B}_{\text{batch}}|} \sum_{(x,y) \in \mathcal{B}_{\text{batch}}} \ell\big(f_\theta(x),\, y\big).$$

where:

• $\theta$: the model parameters
• $\eta$: the learning rate
• $\nabla_\theta$: the gradient with respect to the parameters $\theta$
• $\mathcal{B}_{\text{batch}}$: the mini-batch sampled from the replay buffer $\mathcal{B}$
• $|\mathcal{B}_{\text{batch}}|$: the number of transitions in the mini-batch
• $\ell(f_\theta(x), y)$: the loss on a buffered transition $(x, y)$
• $f_\theta(x)$: the model's prediction for input $x$
• $x, y$: a transition from the buffer, with $x$ the input and $y$ the target
• $\mathcal{B}$: the replay buffer storing past transitions

A replay buffer is a sampled approximation to the retained history. It converts a streaming problem into something closer to a batch problem without giving up the ability to absorb new data, and is one of the main pieces of infrastructure connecting offline and online learning.

Two design choices matter most.

Buffer size controls the effective memory horizon. A small buffer tracks recent dynamics quickly and forgets older ones. A large buffer retains a broad distribution and adapts slowly. This is the same tradeoff as the forgetting factor in RLS, expressed in units of samples rather than a discount rate.

Sampling strategy controls which parts of history are emphasized.

• Uniform sampling treats every stored transition equally. It is simple, unbiased with respect to the buffer contents, and the default in most implementations.
• Prioritized sampling draws transitions with probability proportional to a priority such as the last temporal-difference error or model loss (Schaul et al., 2016). It focuses capacity on the examples the model currently gets wrong. The cost is that the sampling distribution is no longer the data distribution, so importance weights are needed to keep the gradient unbiased.
• Recency-weighted sampling blends the two, up-weighting recent transitions to track drift while still retaining a background of older data.

A replay buffer stores samples and shapes the training distribution. It determines the distribution of data that the model is trained on, and therefore the errors that the model is optimized to reduce. If the buffer is dominated by recent data, the model is optimized for the recent regime. If it contains old data, the model is optimized for a mixture of regimes. The mixture can be desirable when old data remains valid and harmful when it is obsolete.

Replay introduces a second ratio to track alongside the update-to-data ratio: the replay fraction, the proportion of each batch drawn from the buffer rather than from fresh data. A replay fraction of zero is pure streaming. A fraction near one is nearly offline training on a frozen buffer.

Increasing the replay fraction stabilizes learning and reduces forgetting, but it also slows adaptation to real change, because the gradient is dominated by data that may no longer reflect the world. There is no universally correct value. What matters is that the choice is made deliberately and monitored, and that the buffer is large enough to cover the range of conditions the agent is expected to encounter.

The interaction between replay fraction and buffer size can be subtle. A large buffer with a high replay fraction produces a very stable model that changes slowly. A small buffer with a low replay fraction produces a very adaptive model that may forget quickly. A small buffer with a high replay fraction can be worse than either, because the model repeatedly trains on a narrow slice of recent history and can overfit to it while still calling it replay. The design must be considered as a whole.

When the buffer contains transitions generated by older policies, the data are off-policy with respect to the current one. For pure dynamics modeling this often matters less than it does for value learning: in a stationary Markov environment, $p(s_{t+1} \mid s_t, a_t)$ does not depend on which policy selected $a_t$. Data from any policy can therefore train the same conditional dynamics on the state-action support it covers. Hidden state, environment drift, selection effects, or inadequate support break this simple argument.

Coverage is doing real work here. Off-policy data identifies the conditional dynamics only on the region it covers. Outside that region the model is extrapolating, and that is often where the policy goes next.

This asymmetry between dynamics learning and value learning is worth emphasizing. A value function is defined relative to a policy and reward, so naive off-policy targets can be biased; importance sampling, corrected targets, or explicitly off-policy algorithms address that mismatch. A dynamics model is defined by the environment, so data collected under any policy can estimate the same transition conditional when the Markov and stationarity assumptions hold and the state-action pair lies in the data support. Replay is therefore often a natural fit for model-based learning. Its central danger is unsupported extrapolation rather than the policy label attached to an otherwise valid transition.

Catastrophic forgetting is the abrupt loss of previously learned behavior when a neural network is trained on new data. It was documented in connectionist models in the late 1980s (McCloskey and Cohen, 1989; Ratcliff, 1990) and has remained a central obstacle ever since.

The phenomenon is easy to state. Train a network on task A until it performs well. Then train the same network on task B. Performance on task A frequently collapses to near chance, even though the network has more than enough capacity to represent both tasks. The parameters that encoded A were overwritten by the gradients that encode B.

For a world model the consequences are concrete. A model that has learned the dynamics of a pick-and-place task may lose them entirely after being fine-tuned on a new task with a different gripper. A model that has learned how objects fall may lose that knowledge after adapting to a scene where nothing falls. The loss is not graceful degradation; it is erasure.

The word "catastrophic" is chosen deliberately. In many optimization problems, adding new data gently shifts the solution while preserving most of its useful structure. In neural networks trained by gradient descent, adding a new task can shift the solution so sharply that old behavior disappears. The network still has the capacity to represent the old task. The problem is that the parameters have moved to a region of parameter space that represents the new task well and the old task poorly.

Gradient descent has no notion of importance. Every parameter is adjusted in whatever direction reduces the current loss, and a parameter that was critical for task A is treated no differently from one that was never used. When tasks require overlapping parameters but conflicting values, the update for B necessarily damages A.

There is a geometric way to see this. The gradient for task B is a direction in parameter space that decreases $L_B$. Whether it also decreases $L_A$ depends on the angle between the two gradients. When tasks are related, the angle is small and interference is mild. When tasks conflict, the angle is obtuse and progress on one is regression on the other. Catastrophic forgetting is severe gradient interference.

The geometry also explains why forgetting is not always total. If the gradients for two tasks are nearly orthogonal, the update for one has little effect on the other, and the network can learn both without interference. If the gradients are aligned, learning one task may help the other, which is positive transfer. The hard case is when the gradients point in opposite directions on the same parameters. Then any step toward the new task is a step away from the old one.

The underlying tension was named by Grossberg in the 1980s. A learning system must be stable enough to retain old knowledge and plastic enough to acquire new knowledge. These requirements pull in opposite directions.

• Perfect stability means the model never changes, which means it cannot learn anything new.
• Perfect plasticity means the model always changes, which means it retains nothing.

Every continual learning method is a particular point on this spectrum. The engineering question is not whether to be stable or plastic, but where on the spectrum to sit given how fast the world is expected to change and how costly forgetting is expected to be.

The dilemma is not unique to neural networks. Any learning system that updates its parameters in response to new data faces a version of it. What makes neural networks special is the density of their representations: many tasks share parameters, and a single parameter can be important for many behaviors. That sharing is what gives neural networks their generalization power, and it is also what makes them vulnerable to interference.

Methods for mitigating forgetting fall into three broad families. In practice they are combined.

Regularization-based methods penalize changes to parameters that were important for previous tasks. The canonical example is Elastic Weight Consolidation (Kirkpatrick et al., 2017). After training on task A, EWC estimates how sensitive the loss is to each parameter using the diagonal of the Fisher information matrix $F$, and adds a quadratic penalty when training on task B:

$$L(\theta) = L_B(\theta) + \frac{\lambda_{\mathrm{EWC}}}{2}\sum_i F_i\,\big(\theta_i - \theta_{A,i}^{\star}\big)^2,$$

where:

• $L(\theta)$: the total loss while training on task B, combining the task loss with a quadratic penalty
• $L_B(\theta)$: the loss on task B alone
• $\lambda_{\mathrm{EWC}}$: the regularization strength, controlling how strongly the penalty anchors important parameters
• $F_i$: the Fisher information for parameter $i$, measuring how important that parameter was for task A
• $\theta_i$: the current value of parameter $i$
• $\theta_{A,i}^{\star}$: the value of parameter $i$ at the end of task A
• $\theta$: the parameter vector
• $i$: the parameter index

The Fisher value $F_i$ itself is defined as

$$F_i = \mathbb{E}_{(x, y) \sim \mathcal{D}_A}\!\left[\left(\frac{\partial}{\partial \theta_i}\log p_\theta(y \mid x)\right)^{\!2}\right]_{\theta = \theta_A^{\star}},$$

where:

• $(x, y) \sim \mathcal{D}_A$: input-target pairs drawn from the task-A dataset
• $x$: an input (transition) drawn from task A
• $y$: the target (next state) paired with $x$
• $p_\theta(y \mid x)$: the model's predictive distribution over targets given inputs
• $\partial / \partial \theta_i$: the partial derivative with respect to parameter $i$
• $\theta = \theta_A^{\star}$: the parameters frozen at the end of task A
• $\theta$: the model parameters
• $\theta_i$: the $i$-th parameter
• $\mathcal{D}_A$: the task-A dataset

The Fisher value $F_i$ is large for parameters the model relies on and small for parameters it does not. The penalty is therefore a soft, anisotropic spring anchoring important parameters to their old values while leaving unimportant ones free to move. Related methods include Synaptic Intelligence (Zenke et al., 2017) and Memory Aware Synapses (Aljundi et al., 2018), which differ mainly in how they estimate importance.

The key intuition behind EWC is that not all parameters are equally important for a task. Near a fitted probabilistic model, and under the approximations used to derive EWC, the Fisher supplies a local curvature-based importance measure. Penalizing movement in high-Fisher directions can preserve old-task performance while leaving lower-Fisher directions more flexible; it does not guarantee that every direction behaves this way away from the anchor.

Replay-based methods keep a buffer of old examples, or a generative model of them, and mix them into the new task's batches. Gradient Episodic Memory (Lopez-Paz and Ranzato, 2017) goes further by constraining the update using episodic-memory losses from previous tasks. Averaged GEM (Chaudhry et al., 2019) replaces the per-task constraints with one constraint based on an averaged reference gradient, reducing computation and memory.

The projection in GEM is a geometric operation. It takes the current-task gradient and, when necessary, projects it so its first-order inner product with each previous task's episodic-memory gradient is nonnegative. This constrains first-order increases in the aggregate memory loss for each previous task; it is not a per-example or finite-step guarantee. The projected gradient may be smaller than the raw gradient, slowing learning of the new task.

Architectural methods isolate parameters used by different tasks. Progressive Networks (Rusu et al., 2016) freeze old columns and add a new column with lateral connections, so capacity grows while earlier features remain available for transfer. PackNet (Mallya and Lazebnik, 2018) instead iteratively prunes and freezes weights within a fixed network, using task-specific masks and the remaining free capacity. Both protect allocated weights, but their capacity and routing costs differ.

The architectural approach trades capacity and routing complexity for reduced interference. Progressive Networks explicitly use frozen old features through lateral connections and grow with the number of tasks. PackNet keeps model size fixed but consumes free parameters and requires the correct task mask at inference. Protecting parameters prevents direct overwriting of those parameters, though end-to-end retention still depends on routing and evaluation conditions.

To compare methods you need metrics. Let $R_{i,j}$ denote performance on task $j$ after training through task $i$, with $T$ tasks total. Three standard measures are:

$$\begin{aligned} \text{ACC} &= \frac{1}{T}\sum_{j=1}^{T} R_{T,j}, \\ \text{BWT} &= \frac{1}{T-1}\sum_{j=1}^{T-1}\big(R_{T,j} - R_{j,j}\big), \\ \text{FWT} &= \frac{1}{T-1}\sum_{j=2}^{T}\big(R_{j-1,j} - b_j\big). \end{aligned}$$

where:

• $R_{i,j}$: performance on task $j$ after training through task $i$ (so $R_{j,j}$ is performance on task $j$ right after learning it, and $R_{T,j}$ is its performance after all $T$ tasks)

• $T$: the total number of tasks

• $\text{ACC}$: average accuracy across all tasks after training through task $T$

• $\text{BWT}$: backward transfer, the average change in performance on earlier tasks after learning later ones

• $\text{FWT}$: forward transfer, performance on a new task before training it relative to a specified baseline

• $b_j$: baseline performance on task $j$; in the original GEM definition, test performance at random initialization

• $i$: the task index up to which training has proceeded

• $j$: the task index being evaluated

• ACC is final average performance across all tasks. Higher is better.

• BWT, backward transfer, measures how much learning later tasks changed performance on earlier ones. Negative BWT is forgetting. Positive BWT means the later tasks helped.

• FWT, forward transfer, measures how much earlier tasks change performance on a new task before it is trained, relative to $b_j$. In the original GEM metric, $b_j$ is random-initialization performance. Positive FWT indicates beneficial transfer under that baseline.

Note that these metrics are computed on a held-out evaluation set per task, not on the training data. A method that reduces training loss on task A while degrading held-out performance has not avoided forgetting; it has overfit.

The three metrics capture different aspects of continual learning. ACC measures final performance, which is what ultimately matters for deployment. BWT measures whether the model retained old skills. FWT measures whether the model can use old skills to learn new ones faster. A method with high ACC might have poor BWT if the final task is easy and the earlier tasks are irrelevant. A method with high BWT might have low ACC if it is too conservative to learn the new task well. The metrics must be read together, and they must be computed on data that the model has not trained on.

A subtlety that the continual learning literature often understates: forgetting is not always a bug. If the world changed, the old dynamics are no longer true, and a model that insists on them is wrong. The distinction is between forgetting knowledge that is still valid and forgetting knowledge that has become obsolete.

The practical difficulty is that the agent usually cannot tell which is which. A drop in performance on old data could mean the model degraded, or it could mean the old data is no longer representative. Resolving this ambiguity is one of the central problems of continual adaptation, and it is the subject of the next section.

This ambiguity has an important consequence for method design. A method that aggressively prevents forgetting may be preserving obsolete knowledge. A method that aggressively allows forgetting may be discarding valid knowledge. The right balance depends on the environment's rate of change and on the agent's ability to detect change. In practice, the best methods often combine a mechanism for retaining knowledge with a mechanism for detecting when retention is no longer appropriate.

Suppose you accept that the world changes and decide the model should keep learning online. You set up a replay buffer, a modest learning rate, and a periodic update schedule. What could go wrong?

Quite a lot. Online adaptation in a closed loop is a feedback system, and feedback systems have failure modes that open-loop training does not.

The policy is an optimizer over the model's predictions. Optimizers are adversarial to the objects they optimize. If the model has a region where it over-predicts return, the policy will find that region, regardless of how small it is.

This is the optimizer's curse (Smith and Winkler, 2006): selecting the best of many options according to noisy estimates creates an expected optimism bias in the selected estimate. Larger search sets can intensify that selection pressure, but a realized gap need not grow monotonically with every added candidate. In a world model, "many options" means many candidate action sequences, and modern planning methods may consider thousands.

The consequence for online learning can be severe. Data collection concentrates where the model predicts high return. Training on those transitions can correct the error when real outcomes are recorded and weighted adequately. If updates instead rely on model-generated targets, sparse feedback, or a buffer that underrepresents failures, the loop can reinforce an optimistic error.

It is worth distinguishing two sources of error. The first is ordinary approximation error: the model is not a perfect representation of the environment. The second is selection error: the policy chooses actions based on the model's predictions, and selection error amplifies approximation error in the regions where the model is most optimistic. The optimizer's curse is about selection error, not approximation error. Even a model with small average approximation error can have large selection error if its errors are concentrated in the regions the policy prefers.

We use unsafe drift to describe the slow, cumulative movement of a model into a regime where it is confidently wrong and the policy acts on that confidence. It differs from an ordinary performance regression in three ways.

• It is silent. Open-loop prediction error on the training distribution may look fine, because the agent has stopped visiting the region where the model is wrong.
• It is self-reinforcing. Errors steer the policy, and the policy's data confirms the errors.
• It is non-monotonic in the obvious metrics. A model can improve on average loss while becoming less safe, because the average is dominated by easy regions.

Unsafe drift is not merely hypothetical; it is a plausible failure mode for an agent that adapts its model online without adequate external checks. A concrete example appears in the code below: a planner searching over a wider action range than the model was trained on can select extrapolative actions whose model-predicted return exceeds their true return.

The word "drift" suggests slow motion, but the underlying mechanism can be fast. Once the policy discovers a region of model optimism, it will return there repeatedly, and the data from that region will dominate the training signal. The model becomes more confident about its error, which makes the policy more likely to exploit it, which generates more data from the error region. The process can accelerate rather than stabilize.

There is no single technique that eliminates unsafe drift, but there is a well-established set of guardrails that together make it manageable. They fall into four categories.

Constraint-based. Limit how far the model or policy can move per update. Trust-region methods constrain the KL divergence between successive policies; the same idea applies to models through a bound on the change in predicted transition distributions. A small learning rate with a bounded number of updates per unit of new data is the simplest version of this.

Uncertainty-based. Refuse to act on predictions the model is unsure about. Ensembles can disagree where the data is thin, making disagreement a useful heuristic signal, but they can also agree confidently out of distribution. This connects to Stochasticity and Uncertainty: a well-calibrated epistemic uncertainty estimate should reflect lack of knowledge under relevant shifts. Penalizing high estimated uncertainty can deter some unsupported actions.

Verification-based. Require prespecified evidence before a candidate update takes effect. Shadow deployment runs the new model in parallel with the incumbent and compares task, calibration, and safety metrics on the same live inputs without letting the new model act. A mismatch alone is not a reason to reject it; degradation on the chosen criteria is. Rollback checkpoints improve recoverability when monitoring detects degradation, but cannot guarantee that every harmful change will be detected.

Monitoring-based. Track whether live data lies inside the training distribution. Density models, reconstruction error, ensemble disagreement, and simple feature-space distances all provide signals. An out-of-distribution alarm does not fix anything by itself, but it converts a silent failure into a visible one, which is the precondition for any response.

These categories are complementary rather than alternative. Constraint-based methods can limit the size of a change. Uncertainty-based methods can deter actions in poorly supported regions. Verification-based methods can catch some regressions before promotion, while monitoring can reveal failures after deployment. High-consequence systems often layer all four, and the layers are most effective when designed together.

There is a useful reframing that ties these ideas together. Online adaptation is itself a control problem, with the learning process as the plant. The setpoint is stable performance across the conditions the agent encounters. The control inputs are the learning rate, the replay fraction, the update-to-data ratio, and the decision of whether to accept an update at all. The feedback signal is a monitored metric, ideally one that combines open-loop accuracy with closed-loop task performance.

Seen this way, the online learning rate need not be a hyperparameter tuned only once. A practical heuristic is to let monitored surprise influence it: if recent real-outcome prediction error is high relative to a baseline, cautiously raise the learning rate and lower the old-data replay fraction so fresh data has more weight; if error is low and stable, reverse those changes. This is feedback-driven adaptation. It becomes dual control only when the agent also chooses actions for both their control value and the information they reveal, a connection we will examine in Active Perception, Dual Control, and Exploration.

The control view also clarifies why fixed hyperparameters are often a bad fit for continual learning. A fixed learning rate is like a controller with no feedback. It may be tuned for an average rate of change, but it cannot respond to a sudden shift or a long period of stability. A feedback-driven learning rate can. The cost is that the feedback signal must be chosen carefully, because a noisy or misleading signal can cause the controller to oscillate or to adapt in the wrong direction.

It is worth being explicit about what these guardrails do not provide. They do not give formal guarantees that an arbitrary neural world model will remain accurate. They do not detect shift that produces no observable signature. They do not help if the monitoring metric itself is gamed by the same optimization pressure that drives the policy.

What they aim to provide is a reduction in the expected severity of failure and, more importantly, a reduction in the time to detection. In safety-critical deployment, the second is often worth more than the first, because a system that fails visibly and reverts is generally less dangerous than one that fails silently. We return to these questions in depth in Part XII: Reliable World Models.

There is also a deeper limitation: guardrails are designed against known failure modes. Unsafe drift is a known failure mode, so it can be monitored. But an adaptive system can fail in ways that were not anticipated, and those failures may not trigger any of the monitors. The best defense is a combination of principled design, conservative defaults, and human oversight at the points where the system's behavior changes most.

Let us make the tradeoffs quantitative with a scalar example we can solve on paper. Suppose the true relationship is

$$y_t = \theta^\star x_t + \epsilon_t,$$

where:

• $y_t$: the observed target at step $t$
• $\theta^\star$: the true coefficient in the drifting-coefficient example
• $x_t$: the scalar input at step $t$
• $\epsilon_t$: zero-mean observation noise at step $t$
• $\sigma^2$: the variance of the observation noise $\epsilon_t$

with $x_t \sim \mathcal{N}(0, 1)$ and $\epsilon_t \sim \mathcal{N}(0, \sigma^2)$, and suppose $\theta^\star$ changes at some point from one value to another. We want to know how fast SGD tracks the change and how precisely it can localize the new value.

From the derivation above, the bias decays geometrically:

$$\mathbb{E}\big[\theta_t - \theta^\star\big] = \big(1 - \eta \sigma_x^2\big)^t \big(\theta_0 - \theta^\star\big).$$

where:

• $\mathbb{E}[\theta_t - \theta^\star]$: the expected deviation of the estimate from the true parameter after $t$ steps
• $\theta_t$: the parameter estimate after $t$ steps
• $\theta_0$: the initial parameter value
• $\theta^\star$: the true parameter
• $\eta$: the fixed learning rate
• $\sigma_x^2$: the variance of the inputs $x_t$
• $t$: the number of steps elapsed, so $\eta \sigma_x^2$ is the per-step contraction factor

With $\sigma_x^2 = 1$ and $\eta = 0.02$, the decay rate is $0.98$ per step. To reduce the initial error by a factor of $e$ takes about $1/0.02 = 50$ steps. Here is where that number comes from. We want the smallest $t$ with $(1 - \eta\sigma_x^2)^t \le e^{-1}$. Taking logarithms,

$$\begin{aligned} t \ln(1 - \eta\sigma_x^2) &\le -1 && \text{(apply } \ln \text{)} \\ t &\ge \frac{-1}{\ln(1 - \eta\sigma_x^2)} && \text{(divide by the negative log)} \\ &\approx \frac{1}{\eta\sigma_x^2} && \text{(use } \ln(1-z) \approx -z \text{ for small } z\text{)} \end{aligned}$$

where:

• $t$: the number of steps
• $\eta$: the learning rate
• $\sigma_x^2$: the variance of the inputs $x_t$
• $z$: a small positive quantity, here $z = \eta\sigma_x^2$

With $\eta\sigma_x^2 = 0.02$ this gives $t \approx 50$ steps. That is the adaptation time.

The adaptation time is the first half of the tradeoff. It tells us how long the estimator takes to forget an old value after the world changes. The second half is the noise ball, which tells us how precisely the estimator can settle on the new value once it has arrived.

Now the noise. At stationarity, the variance satisfies

$$\mathrm{Var}(\theta_\infty) = \frac{\eta^2 \sigma_x^2 \sigma^2}{1 - \mathbb{E}\big[(1 - \eta x_t^2)^2\big]}.$$

where:

• $\mathrm{Var}(\theta_\infty)$: the stationary variance of the estimate around the true parameter
• $\theta_\infty$: the stationary parameter estimate around which the variance is measured
• $\eta$: the learning rate
• $\sigma_x^2$: the variance of the inputs $x_t$
• $x_t$: the scalar input at step $t$
• $\sigma^2$: the variance of the observation noise $\epsilon_t$
• $\mathbb{E}\big[(1 - \eta x_t^2)^2\big]$: the expected squared contraction factor, which appears in the denominator

Replacing the random contraction by its mean gives the moment-closure approximation $\mathbb{E}\big[(1 - \eta x_t^2)^2\big] \approx (1 - \eta\sigma_x^2)^2$. Under that approximation,

$$\begin{aligned} \mathrm{Var}(\theta_\infty) &= \frac{\eta^2 \sigma_x^2 \sigma^2}{1 - (1 - \eta\sigma_x^2)^2} \\ &= \frac{\eta\,\sigma^2}{2 - \eta\,\sigma_x^2} && \text{(expand the denominator)} \\ &\approx \frac{\eta\,\sigma^2}{2} && \text{(small } \eta \text{)} \end{aligned}$$

where:

• $\mathrm{Var}(\theta_\infty)$: the stationary variance of the estimate around the true parameter
• $\theta_\infty$: the stationary parameter estimate around which the variance is measured
• $\eta$: the learning rate
• $\sigma_x^2$: the variance of the inputs
• $\sigma^2$: the variance of the observation noise $\epsilon_t$

This is the noise ball: the residual fluctuation of the estimate around the true value. Its radius is

$$\sqrt{\mathrm{Var}(\theta_\infty)} \approx \sqrt{\frac{\eta\,\sigma^2}{2}}.$$

where:

• $\sqrt{\mathrm{Var}(\theta_\infty)}$: the standard deviation of the stationary estimate fluctuation, also called the noise ball radius
• $\mathrm{Var}(\theta_\infty)$: the stationary variance of the estimate around the true parameter
• $\theta_\infty$: the stationary parameter estimate around which the variance is measured
• $\eta$: the learning rate
• $\sigma^2$: the observation noise variance

With $\eta = 0.02$ and $\sigma = 0.05$, the approximate stationary standard deviation is about $0.005$. That is the scale of an instantaneous stationary fluctuation, not a hard detection limit: averaging or sequential tests can detect smaller persistent shifts. A change of $0.3$ is much larger, and the roughly 50-step e-folding time means the estimate moves substantially toward it over tens of steps rather than settling immediately.

The tradeoff is now explicit. Increasing $\eta$ shortens the adaptation time and enlarges the noise ball. Decreasing $\eta$ does the reverse. There is no setting that makes both small; you are choosing where on the curve to sit. This is the same tradeoff that a replay buffer expresses in sample units, and the same tradeoff that a forgetting factor expresses as a discount rate. All three are parametrizations of a single decision about how much history to retain.

The figure below makes that frontier visible. Each point on the curve corresponds to one learning rate, with the adaptation time on the horizontal axis and the noise ball radius on the vertical axis.

One more observation from the variance formula. The noise ball shrinks only as $\sqrt{\eta}$, so halving the residual fluctuation requires quartering the learning rate, which quadruples the adaptation time. The returns to precision are poor. If you need a precise estimate, gradient descent with a fixed learning rate is the wrong tool; you want a method that reduces its effective step size as evidence accumulates, which is exactly what RLS does until it is given a forgetting factor.

The worked example also shows why the choice of learning rate depends on the scale of the change you expect. If the true coefficient changes by 0.3, a learning rate of 0.02 tracks it quickly. If the true coefficient changes by 0.003, that same learning rate cannot distinguish the change from noise. A practical system should therefore relate its learning rate to the smallest change it needs to detect and the fastest change it needs to track. Those two requirements pull in opposite directions, and the resolution is usually to use a larger learning rate for adaptation and a smaller one for fine-tuning, or to use a method that adapts the learning rate itself.

We will now build three small experiments that isolate the mechanisms discussed above. The first trains a small neural dynamics model on two environments in sequence, comparing naive fine-tuning against replay and against Fisher-weighted regularization. The second tracks a drifting coefficient with and without a replay buffer. The third demonstrates model exploitation in a planner. Each experiment precomputes its results from variables and is followed by a discussion of what the numbers mean.

The experiments are deliberately small. The goal is not to achieve state-of-the-art performance but to make the mechanisms visible. The first experiment uses a two-environment sequence so that forgetting and retention can be measured directly. The second uses a scalar drifting coefficient so that the adaptation and noise tradeoff can be seen in a single trace. The third uses a planner with a widening search budget so that the gap between predicted and true return can be measured. Each experiment isolates one part of the problem.

We start with imports and a seeded random generator. Everything downstream uses this generator so results are reproducible.