The World-Model Evaluation Ladder › Open-loop rollout

An open-loop rollout feeds predicted states back into the model while using a fixed action sequence. This tests the composed transition map, including how errors at one input affect later predictions. It does not imply that the one-step evaluation samples were independent.

First distinguish two environments. In a stochastic environment, $s_{t+1}\sim T(\cdot\mid s_t,a_t)$. For the pathwise bound below, restrict attention to deterministic dynamics with a state-valued map $F$:

$$s_{t+1}=F(s_t,a_t),\qquad \hat s_0=s_0,\qquad \hat s_{t+1}=f_\theta(\hat s_t,a_t).$$

Where:

• $F$: deterministic true dynamics, returning a successor state;
• $T$: a transition probability kernel; in this deterministic special case it places all probability at $F(s,a)$;
• $s_t$ and $\hat s_t$: the true and predicted states;
• $a_t$: the common action at step $t$;
• $H$: the positive integer rollout horizon.

Both branches use the same initial state and action sequence $a_{0:H-1}$, with no observation corrections during prediction. Trajectory RMSE and endpoint error summarize different parts of the path:

$$E_H(f_\theta)=\sqrt{\frac1H\sum_{t=1}^H\|\hat s_t-s_t\|^2}, \qquad E_{\mathrm{final}}(f_\theta)=\|\hat s_H-s_H\|.$$

The RMSE averages the $H$ successor states, excluding the common zero-error initial state. Endpoint error scores only the final state. Neither dominates as a metric: an application that needs the whole path may need intermediate errors, while terminal decisions may prioritize the endpoint. For stochastic rollouts, state whether the score compares distributions, expected quantities, or coupled sampled paths; unrelated noise draws can contribute to a pathwise difference.

To derive an error envelope, let $d_t=\|\hat s_t-s_t\|$ and assume:

• common fixed actions and a common initial state;
• a uniform $L\geq0$ satisfying $\|f_\theta(x,a)-f_\theta(y,a)\|\leq L\|x-y\|$ for every compared true/predicted input pair and action along the trajectories;
• a uniform local error bound $\|f_\theta(s,a)-F(s,a)\|\leq e_1$ on every true input encountered.

A sufficient stronger condition is that both bounds hold throughout a reachable region containing those trajectories. Add and subtract $f_\theta(s_t,a_t)$, then use the triangle inequality:

$$\begin{aligned} d_{t+1} &\leq\|f_\theta(\hat s_t,a_t)-f_\theta(s_t,a_t)\| +\|f_\theta(s_t,a_t)-F(s_t,a_t)\|\\ &\leq Ld_t+e_1. \end{aligned}$$

Here $d_0=0$, $L$ bounds amplification in the stated norm, and $e_1$ bounds local transition error. Iterating this inequality yields

$$d_H\leq e_1\sum_{k=0}^{H-1}L^k= \begin{cases} e_1\dfrac{L^H-1}{L-1},&L>1,\\[4pt] He_1,&L=1,\\[4pt] e_1\dfrac{1-L^H}{1-L},&0\leq L<1. \end{cases}$$

These are upper envelopes, not observed growth laws. If the same uniform assumptions hold for every horizon and $L<1$, the envelope does not exceed $e_1/(1-L)$. At $L=1$ it is linear, and at $L>1$ it is exponential. Actual errors can remain zero even under a loose bound with $L>1$.

The distinction matters in three ways.

• Stability: one-step average loss does not measure the amplification of perturbations. For differentiable maps, a suitable uniform bound on the induced Jacobian norm can establish the Lipschitz condition on an appropriate region. A spectral radius at one state is not a general bound on products of changing Jacobians.
• Horizon: report the horizon and the trajectory statistic together. The same models can change order as the horizon changes.
• Ranking: similar held-out average risks do not determine rollout ranking or supply the uniform $e_1$ used above. A contractive model can sometimes outperform a less biased expansive model in rollout, but neither stability nor local accuracy alone fixes the ranking.

The following plot holds $e_1=0.01$ fixed and draws the envelope for three chosen values of $L$. It plots the bound, not measured trajectory errors.

The envelope assumes accuracy at the inputs reached by the rollout. A held-out average error under $D$ does not establish that condition. Predicted trajectories may enter regions with few observed samples, or regions outside the support of $D$ altogether. These are different cases: a finite sample can omit inputs that still have positive population probability.

This mismatch between observed training inputs and model-generated inputs is one mechanism behind compounding-error problems. It motivates training methods that explicitly confront model-generated states, such as Talvitie's self-correcting models. The paper analyzes that mechanism under its stated assumptions; it does not imply that every high-capacity model fails at long horizons. [Part VII, Ch 4: Policies Learned in Imagination] is planned to cover the planning use of such rollouts.