Online, Offline, and Continual Learning › Summary

• Batch, streaming, and continual learning are regimes distinguished by how data reaches the optimizer, not by differences in the optimizer itself. Batch training is reproducible but stale; streaming training is adaptive but unstable; continual learning must serve a sequence of regimes without losing earlier ones.
• The learning rate is a memory knob. For a scalar linear model, the bias decays at rate $1 - \eta\sigma_x^2$ and the stationary noise ball has radius approximately $\sqrt{\eta\sigma^2/2}$. Larger learning rates adapt faster and fluctuate more; the returns to precision are poor because the noise ball shrinks only as $\sqrt{\eta}$.
• Recursive least squares and the Kalman filter give exact streaming estimators for linear models, and exponential forgetting discounts old information. With persistent excitation and no forgetting, shrinking covariance makes adaptation progressively slower.
• Distribution shift has several useful taxonomies. In the three-part taxonomy used here, a changing policy can induce covariate shift, changed transition dynamics are a form of concept shift, and changes in task or goal frequencies can resemble prior-probability shift.
• Replay is the standard remedy, converting a streaming problem into an approximate batch problem by sampling mini-batches from a stored history. Buffer size and sampling strategy control the memory horizon, and the replay fraction controls the stability-plasticity balance.
• Catastrophic forgetting is gradient interference. Parameters that encode one task are overwritten by gradients that encode another, and the phenomenon is severe because gradient descent has no notion of parameter importance.
• Three families of methods mitigate it: regularization such as EWC, which anchors important parameters using the Fisher information; replay, which reintroduces old data; and architectural isolation, which allocates separate parameters per task.
• Metrics matter. Average accuracy, backward transfer, and forward transfer measure different things, and all three must be computed on held-out data. A method that reduces training loss while degrading held-out performance has not avoided forgetting.
• Unsafe drift is a closed-loop failure mode of online adaptation. The policy optimizes model predictions, so selection can amplify optimistic model errors. The planning experiment illustrates this expected selection bias without claiming a monotonic guarantee for every candidate set.
• Guardrails reduce risk but do not eliminate it. Constraint-based, uncertainty-based, verification-based, and monitoring-based mechanisms together make failure more visible and more recoverable. They do not make it impossible.
• Adaptation is a control problem. Learning rate, replay fraction, update-to-data ratio, and update acceptance are control signals that should respond to detected change rather than being fixed hyperparameters. We will see this reframing again when we discuss exploration and dual control in Part VII.

The next chapter, Pretraining, Post-Training, Scaling, and Adaptation, takes these mechanisms and asks how they compose at the scale of modern foundation models, where the same questions about staleness, forgetting, and drift appear in a different guise: what to freeze, what to fine-tune, and how to adapt a very large model to a new domain without destroying what it already knows.