Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Online, Offline, and Continual Learning › Batch and Streaming Updates

Michael BrenndoerferJuly 5, 202615 min read

World Models HandbookAll chapters

Part 6 of 12Learning World Models at Scale

Chapter 35Online, Offline, and Continual Learning › Batch and Streaming Updates

Reading section 2 of 11 · Batch and Streaming Updates · 15 minSection 2 of 11 · 15 min

Section 2 of 11 · 15 min read

Online, Offline, and Continual Learning › Batch and Streaming UpdatesOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The distinction between batch and streaming updates is about how data reaches the optimizer, not about how the optimizer works.

In the batch (or offline) setting you have a dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ collected beforehand. You choose parameters by minimizing the empirical risk

\hat{\theta} = \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} \ell\big(f_\theta(x_i),\, y_i\big).

where:

$\hat{\theta}$ : the parameter estimate that minimizes the average loss over the dataset
$\theta$ : the parameters being optimized
$\mathcal{D}$ : the dataset of input-target pairs, $\{(x_i, y_i)\}_{i=1}^N$
$N$ : the number of examples in the dataset $\mathcal{D}$
$\ell(\cdot, \cdot)$ : the per-example loss function, such as squared error
$f_\theta(x_i)$ : the model's prediction for input $x_i$
$y_i$ : the target value paired with $x_i$
$x_i$ : the $i$ -th input example

For a world model, $x_i$ is a transition input such as $(s_t, a_t)$ , $y_i$ is the target such as $s_{t+1}$ , and $f_\theta$ is the learned dynamics. This is the setting of Part V and Part VI so far. It has several properties worth naming explicitly:

Reproducibility. The result can be reproduced when the dataset, architecture, optimization procedure, software stack, hardware, and random seeds are controlled.
Multiple passes. You can shuffle and revisit every example as many times as you like.
Auditability. You can inspect exactly what the model was trained on.
Staleness. Once training stops, the model has no mechanism to notice that the world changed.

These properties are tightly connected. Reproducibility comes from the fact that the dataset is fixed, so the same optimization procedure applied to the same data yields the same result up to random seeds and hardware nondeterminism. Multiple passes are possible because the dataset is stored in full and can be revisited indefinitely; this is what allows offline training to extract many gradient steps per example. Auditability follows from the same storage: the training distribution is not a moving target but a concrete artifact that can be inspected, versioned, and checked for biases or gaps. Staleness is the price. A model that was trained on a fixed dataset has no internal signal that the world has moved on. Its errors on the new world are not necessarily large, but they are unmonitored.

In the streaming (or online) setting data arrives as a sequence $(x_1, y_1), (x_2, y_2), \ldots$ and you update after each arrival:

\theta_{t+1} = \theta_t - \eta_t \,\nabla_\theta\, \ell\big(f_{\theta_t}(x_t),\, y_t\big).

where:

$\theta_{t+1}$ : the parameters after processing the $t$ -th example
$\theta_t$ : the parameters before the update
$\eta_t$ : the learning rate at step $t$
$t$ : the current step index in the stream
$x_t, y_t$ : the $t$ -th input and target pair arriving from the stream
$\ell(\cdot, \cdot)$ : the per-example loss function
$\nabla_\theta \ell(f_{\theta_t}(x_t), y_t)$ : the gradient of the loss on the current example with respect to the parameters
$f_{\theta_t}(x_t)$ : the model's prediction for input $x_t$ using the current parameters
$\theta$ : the model parameters

This is stochastic approximation. Under the usual regularity assumptions on the objective, gradients, and noise, a standard route to convergence uses the classical Robbins--Monro step-size conditions. They require that the learning rates accumulate enough total step size to keep moving while the squared learning rates accumulate little enough total noise for it to average out:

\begin{aligned} \sum_{t=1}^{\infty} \eta_t &= \infty && \text{(step sizes must not shrink too fast)} \\ \sum_{t=1}^{\infty} \eta_t^2 &< \infty && \text{(squared step sizes must shrink fast enough)} \end{aligned}

where:

$\eta_t$ : the learning rate applied at step $t$
$\sum_{t=1}^{\infty} \eta_t$ : the total accumulated step size, which must diverge so the estimator can travel any distance
$\sum_{t=1}^{\infty} \eta_t^2$ : the total accumulated squared step size, which must converge so gradient noise averages out

Here $\eta_t$ is the learning rate applied at step $t$ . Such schedules are called square-summable but not summable: the first condition keeps the estimator moving in expectation, while the second keeps its variance from accumulating.

The first condition says the learning rate must not shrink so fast that the model freezes before it arrives. The second says it must shrink fast enough that accumulated gradient noise averages out. A schedule like $\eta_t = 1/t$ satisfies both. The schedule $1/\sqrt{t}$ and a positive constant both violate square summability; the constant still satisfies the divergent-sum condition. That persistent step size is useful in non-stationary settings because it keeps the estimator responsive, at the cost of a nonzero noise floor.

The schedule shape determines how the estimator behaves over time. The figure below compares three common choices on logarithmic axes, where the difference between square-summable and non-summable decay is easiest to read.

In[3]:

Code

import numpy as np

schedule_t = np.arange(1, 201)
eta_inverse_t = 1.0 / schedule_t
eta_inverse_sqrt = 1.0 / np.sqrt(schedule_t)
eta_constant = np.full_like(schedule_t, 0.05, dtype=float)

Out[4]:

Visualization

Log-log plot of inverse-t, inverse-sqrt-t, and constant learning rate schedules. — Three learning-rate schedules over 200 streaming steps on logarithmic axes. The inverse schedule 1/t satisfies both Robbins--Monro sum conditions. The inverse-square-root schedule decays more slowly but is not square-summable, while the constant schedule never decays.

The Robbins and Monro conditions are a useful reference point even when they do not literally apply. They describe the behavior of an estimator that is trying to converge to a fixed target in a stationary world. In a world-modeling loop, the target is not fixed: the true dynamics may change, and even if they do not, the distribution of states and actions that the model sees depends on the policy, which depends on the model. The classical conditions therefore tell us what convergence would require if the world stood still. They also tell us why practical systems use constant or slowly decaying learning rates: the agent is not trying to converge to a fixed answer, because the answer keeps moving.

A useful mental model is to separate two questions. The first is whether the optimizer can reach a good solution if one exists. The second is whether the optimizer can track a solution that changes over time. Batch learning is optimized for the first question. Streaming learning with a constant learning rate is optimized for the second. Continual learning must answer both, often without knowing which regime it is currently in.

Batch and streaming are endpoints of a spectrum. Full-batch gradient descent uses all $N$ examples per step. Mini-batch SGD uses $B$ examples per step. Pure online SGD uses one. Everything between is a design choice about how much of the historical distribution to include in each update. A replay buffer, which we turn to in the next section, is the device that lets you move along this spectrum even when data arrives one sample at a time.

The spectrum covers both batch size at a single step and how long data is retained. A method could update on a mini-batch of recent data and still be highly online if that mini-batch is discarded immediately. Another method could update on a single example drawn from a large historical buffer and behave much more like batch training. The relevant quantity is the effective age distribution of the data contributing to each update.

The learning rate as a memory knobLink Copied

There is a precise sense in which the learning rate controls how much history the estimator remembers. Consider the simplest case: a scalar linear model $y_t = \theta^\star x_t + \epsilon_t$ with $\epsilon_t$ zero-mean and independent of $x_t$ . The SGD update is

\theta_{t+1} = \theta_t - \eta\,(\theta_t x_t - y_t)\,x_t.

where:

$\theta_{t+1}$ : the updated parameter estimate
$\theta_t$ : the current parameter estimate
$\eta$ : the constant learning rate
$x_t$ : the scalar input at step $t$
$y_t$ : the observed target at step $t$
$\theta_t x_t - y_t$ : the prediction error, which drives the update

Here $\theta_t$ is the current parameter estimate, $\eta$ is the constant learning rate, $x_t$ is the scalar input at step $t$ , $y_t$ is the observed target, and $\theta_t x_t - y_t$ is the prediction error that drives the update.

Writing the error as $e_t = \theta_t - \theta^\star$ and substituting $y_t$ gives

e_{t+1} = \big(1 - \eta x_t^2\big)\, e_t + \eta x_t \epsilon_t.

where:

$e_{t+1}$ : the parameter error after the update, $\theta_{t+1} - \theta^\star$
$e_t$ : the parameter error before the update, $\theta_t - \theta^\star$
$\eta$ : the learning rate
$x_t$ : the scalar input at step $t$
$\epsilon_t$ : the zero-mean observation noise at step $t$
$\theta_t$ : the parameter estimate before the update
$\theta_{t+1}$ : the parameter estimate after the update
$\theta^\star$ : the true parameter value

The first term $(1 - \eta x_t^2) e_t$ shrinks the existing error, while the second term $\eta x_t \epsilon_t$ injects noise. Take expectations over the noise and inputs, assuming the current sample is independent of $e_t$ , $\mathbb{E}[x_t\epsilon_t]=0$ , and $\mathbb{E}[x_t^2]=\sigma_x^2$ :

\mathbb{E}[e_{t+1}] = \big(1 - \eta \sigma_x^2\big)\,\mathbb{E}[e_t].

where:

$\mathbb{E}[e_t]$ : the expected parameter error at step $t$
$\eta$ : the learning rate
$\sigma_x^2$ : the variance of the inputs $x_t$
$1 - \eta \sigma_x^2$ : the geometric decay rate of the bias
$\mathbb{E}[e_{t+1}]$ : the expected parameter error at step $t+1$

The bias decays geometrically at rate $1 - \eta\sigma_x^2$ . A large learning rate forgets old errors quickly. A small one holds onto them. We will return to this in the worked example.

This recursion is the simplest possible model of memory in gradient descent. The current error is a weighted combination of the previous error and a noise term. The weight on the previous error is approximately one minus the learning rate times the input variance. If that weight is close to one, the estimator remembers its old error for a long time. If it is close to zero, the estimator discards its old error almost immediately. In a stationary world, a small weight is good, because the old error was itself caused by noise or by initialization and should be forgotten. In a non-stationary world, the old error may contain information about dynamics that are still partially valid, and forgetting it completely may be wasteful.

There is a second sense in which the learning rate controls memory. The stationary variance of the estimator is proportional to the learning rate. A large learning rate makes the estimator responsive but noisy. A small learning rate makes it precise but slow. The memory knob and the noise knob are the same knob, turned in opposite directions. This is the fundamental tradeoff that reappears in every method in this chapter.

Recursive least squares and the Kalman connectionLink Copied

For linear models there is an exact streaming estimator that avoids gradient descent entirely. Recursive least squares (RLS) maintains a parameter estimate $\theta_t$ and a covariance matrix $P_t$ :

\begin{aligned} P_t &= P_{t-1} - \frac{P_{t-1} \phi_t \phi_t^\top P_{t-1}}{1 + \phi_t^\top P_{t-1} \phi_t}, \\ \theta_t &= \theta_{t-1} + P_t \phi_t \big(y_t - \phi_t^\top \theta_{t-1}\big). \end{aligned}

where:

$P_t$ : the inverse information matrix after step $t$ ; under a matching linear-Gaussian model with normalized observation noise, it is proportional to the posterior covariance of $\theta$
$P_{t-1}$ : the covariance before the update
$\phi_t$ : the input (feature) vector at step $t$
$\theta_t$ : the parameter estimate after step $t$
$\theta_{t-1}$ : the parameter estimate before the update
$y_t$ : the target at step $t$
$y_t - \phi_t^\top \theta_{t-1}$ : the prediction residual, which drives the parameter correction
$\theta$ : the parameter vector being estimated

If you have read Bayesian Filtering and Belief States, this should look familiar: RLS is a Kalman filter for a constant parameter with no process noise. Under a matching linear-Gaussian model, $P_t$ is proportional to the posterior covariance of $\theta$ , and it shrinks as evidence accumulates. With persistent excitation in every parameter direction, $P_t$ approaches zero and the gain becomes arbitrarily small. Plain RLS then adapts increasingly slowly to a change; directions that are not sufficiently excited need not collapse.

The intuition is that RLS is not merely averaging recent samples. It is maintaining a full second-order description of what it knows. When data is plentiful and the model is confident, the covariance becomes small, so the update gain becomes small. When data is scarce or the input directions are poorly explored, the covariance remains large, so the estimator is willing to move. This is exactly the behavior a learner should have in a stationary world. It is exactly the wrong behavior in a world that changes, because the estimator becomes most confident precisely when it has seen the most data, which may be the data from before the change.

The fix is to add process noise, which produces a forgetting factor $\lambda$ :

P_t = \frac{1}{\lambda}\left(P_{t-1} - \frac{P_{t-1} \phi_t \phi_t^\top P_{t-1}}{\lambda + \phi_t^\top P_{t-1} \phi_t}\right).

where:

$\lambda$ : the forgetting factor, with $0 < \lambda \le 1$ ; smaller $\lambda$ down-weights older data more aggressively and inflates the covariance to preserve plasticity
$P_t, P_{t-1}$ : the covariance after and before the update
$\phi_t$ : the input vector at step $t$

With $\lambda < 1$ , older observations are exponentially down-weighted with time constant $1/(1-\lambda)$ . This is the cleanest formal statement of the memory-versus-plasticity tradeoff: the forgetting factor is a literal discount on history. A value of $\lambda = 0.99$ means the effective window is roughly 100 samples; $\lambda = 0.999$ means roughly 1000.

The forgetting factor has a statistical interpretation: it inflates the covariance carried from the previous step, expressing a belief that old information is becoming less relevant. In a Kalman interpretation this resembles process noise, although exponential forgetting generally implies a state-dependent covariance inflation rather than a fixed random-walk covariance. In a world model, the drifting quantity might be a physical coefficient, a reward function, or an entire latent dynamics model.

Update-to-data ratioLink Copied

A quantity that matters in every model-based agent is the update-to-data ratio, sometimes called the replay ratio: how many gradient steps the model takes per environment step collected. In offline training the ratio is enormous (many epochs over a fixed dataset). In a purely streaming loop it is one. In practice, agents such as Dreamer and MBPO sit somewhere in between, and the choice has real consequences. A high ratio extracts more information per real interaction, which is the entire appeal of learning a model in the first place. But a high ratio also means the model is repeatedly optimized against a fixed, possibly stale dataset, so it overfits to whatever the last batch of data happened to contain. Modern practice typically sets an explicit update budget and monitors both open-loop and closed-loop metrics against it.

The update-to-data ratio is one of the few quantities in model-based learning that directly connects computational budget to statistical behavior. If the ratio is one, the model sees each transition exactly once and then moves on. If the ratio is large, the model may see the same transition many times, which can be useful when real interaction is expensive but dangerous when the transition is no longer representative. A high ratio also changes the effective learning rate schedule: many small updates on the same data can approximate a larger update, and many updates on stale data can drive the model to memorize idiosyncrasies of the buffer.

Assumptions that quietly breakLink Copied

The streaming update above assumes the data are drawn from a stationary distribution and that consecutive samples are independent. Neither assumption holds in a world-modeling loop.

Stationarity fails whenever the environment changes, the agent's policy changes, or the agent's own learning changes what it does.
Independence fails because consecutive transitions in a trajectory are highly correlated. A robot that has just turned left is likely to turn left again.

Correlated samples make the gradient an average over a stretch of trajectory rather than over the distribution of interest, which biases updates and inflates effective variance. This is one reason practical systems shuffle, buffer, and batch rather than truly updating one sample at a time. It is also why the word "online" in modern deep RL usually means "interleaved with data collection," not "one gradient step per sample with no memory."

There is a further assumption that is often invisible: the loss being minimized is assumed to be the right objective for the deployment distribution. In a stationary supervised problem, minimizing average loss on the training distribution is a reasonable proxy for minimizing loss on the test distribution. In a world model, the training distribution is generated by the agent's own behavior, and the test distribution is whatever distribution the agent will encounter in the future. Those two distributions can diverge dramatically. Minimizing loss on yesterday's data may not reduce loss on tomorrow's data if the policy has moved on.