Online, Offline, and Continual Learning › Online, Offline, and Continual Learning

A warehouse robot spends its first month learning how boxes slide when pushed. The model is good. Then the facility installs a new conveyor belt, and the same pushes produce different outcomes. The robot's world model is now wrong, and it has no way of knowing that unless it keeps learning. If it keeps learning, it risks overwriting the month of experience it accumulated. And if it adapts too eagerly to the new belt, it may become worse at everything else.

This chapter is about that tension. Every world model we have built in this handbook was trained by minimizing some loss on some dataset, as we saw in Predictive and Generative Objectives and Data Curation and Synthetic Data. But real agents do not stop collecting data when training ends. They keep acting, keep observing, and keep receiving new transitions that may or may not resemble the ones they were trained on. The question is how to keep the model useful over time.

That question changes what a world model is. A model trained once on a fixed dataset is a snapshot of the world as it was during data collection. A model that keeps learning is a process that must decide, at every update, how much to trust the past and how much to trust the present. The design choices that govern that decision determine whether the agent accumulates competence, remains frozen in an obsolete regime, or chases its own errors into regions where it is confidently wrong.

We will work through three regimes and the problems that separate them.

• Offline learning fits a model once to a fixed dataset. It is reproducible, easy to evaluate, and easy to audit.
• Online learning interleaves updates with data collection. It is adaptive, but every update changes the data the agent will collect next.
• Continual learning faces a sequence of tasks, environments, or regimes and must perform well on all of them, not only the most recent one.

Each regime brings its own failure mode. Offline models go stale. Online models drift. Continual models forget. These are not three separate problems to be patched independently; they are three faces of a single design decision about how much weight the model should give to old data relative to new data. That decision is the subject of this chapter.

The regimes also differ in what information is available at training time. In the offline regime, the dataset is fixed before optimization begins, so every update can be evaluated against a stable objective. In the online regime, the objective itself is moving because the data distribution is generated by a policy that the model is helping to change. In the continual regime, there may be no single objective at all, only a sequence of objectives that may conflict. Each transition between regimes removes an assumption that made the previous regime easier to analyze.

We will also keep a distinction in view that runs through the whole book. A world model can be evaluated in two very different ways. Open-loop accuracy asks whether predicted futures match observed futures. Closed-loop usefulness asks whether an agent that plans with the model achieves its goals. A model can improve on the first metric while degrading on the second, because the policy is an optimizer and optimizers find the model's mistakes. Any discussion of online adaptation has to hold both metrics at once.

That distinction is worth making concrete early. Open-loop accuracy is measured by holding the model fixed, feeding it a sequence of inputs from a held-out dataset, and comparing its predictions to the recorded targets. Closed-loop usefulness is measured by placing the model inside an agent, letting the agent choose actions, and evaluating the outcomes in the real environment. The first metric treats the model as a predictor. The second treats it as a component whose errors alter the distribution of future inputs. A model can be excellent as a predictor on the data it has seen and catastrophic as a component, because the agent will deliberately steer toward the parts of the input space where the model's predictions are most favorable, not necessarily most accurate.

The definition above contains a subtle phrase: "presented sequentially." A continual learning problem is more than a collection of datasets because the order matters. If the agent sees task A and then task B, the update for task B can erase task A. If it sees them interleaved, the interference may be milder. If it sees task A, then task B, then task A again, it may recover some performance on A, but only by damaging B. Continual learning is therefore a study of trajectories through parameter space, not only final loss values on individual tasks.