PILCO, PETS, Ensembles, and MBPO › Short model rollouts for policy optimization

Original MBPO combines:

• A probabilistic dynamics-and-reward ensemble. In the original MBPO paper, each member predicts next state and reward conditioned on current state and action. At each simulated transition, the algorithm samples a model from the ensemble; disagreement is not used as an explicit stopping rule or caution penalty.
• A short-branch generator. Branch starts are sampled from the real-environment replay buffer. The current policy supplies actions for model branches up to a globally configured rollout cap $k$, and generated transitions enter a model replay buffer.
• An off-policy learner. In the released implementation, SAC updates its actor and Q-functions from minibatches containing both real and model-generated replay data.

The released MBPO model is trained to predict reward and a state increment. Before sampling, the released simulator adds the current state to each ensemble member's predicted increment mean; the sampled vector is then split into reward and next state. A task-specific termination function supplies the done flag. The resulting synthetic replay entry contains $(s,a,r,s',\mathrm{done})$; termination is not predicted by an extra model head.

Sampling across ensemble members can diversify generated transitions when their predictive distributions differ, but baseline MBPO does not infer a locally safe horizon from ensemble spread, reject uncertain transitions, or make SAC intrinsically cautious. Those would be additional algorithmic mechanisms.

The paper's analysis compares true return with return under $k$-step branches. Its first worst-case bound uses model error measured under the data-collecting policy together with divergence $\epsilon_\pi$ between that policy and the updated policy. The paper explicitly describes this version as overly pessimistic: taken literally, it favors real off-policy data over model-generated data and does not justify a positive rollout length.

The paper's refined analysis aims to estimate model error $\epsilon'_m$ on the updated policy's state-action distribution. The expression printed in Theorem 4.3 has two qualitative dependencies on $k$:

• policy-divergence contributions decrease geometrically with $\gamma^k$, where $\gamma$ is the discount factor;
• the current-policy model-error contribution grows linearly with $k\epsilon'_m$.

All terms are scaled by reward and discount-dependent factors. For this printed penalty, sufficiently small $\epsilon'_m$ can make a positive finite $k$ preferable to $k=0$; that algebra does not show that actual policy return must have an interior optimum. Lemma B.4 in the full paper with appendices defines pre- and post-branch errors but applies them to reversed time ranges in its proof, a mismatch also identified by an independent reproduction. The main Theorem 4.3 statement also describes updated-policy model error while indexing the displayed condition by the data-collecting policy; Appendix A.2 uses the updated policy. We therefore treat the printed tradeoff as the paper's motivation for short model use, not an independently verified performance guarantee. Model error on the current policy distribution still has to be estimated, and rollout schedules require empirical validation.

Original MBPO uses a rollout length scheduled globally by training epoch, not a state-dependent horizon chosen from local disagreement. Every branch generated at a given training stage uses that configured $k$ as a maximum; terminal branches can end sooner. Ensemble spread remains a heuristic disagreement statistic; using it for adaptive local horizons would require a separately specified and validated method.

The algorithm maintains a real replay buffer and a model replay buffer. In the released implementation, policy minibatches mix both sources using a configured real-data fraction. The paper's Algorithm 2 pseudocode, by contrast, writes policy updates on $\mathcal{D}_{\mathrm{model}}$ alone. For example, a released HalfCheetah configuration fixes the real-data fraction at 0.05 within a run, while the MBPO class separately defaults to 0.1 if no override is supplied; model-generated samples form most of the minibatch in either case. Meanwhile, the rollout length may increase on a global schedule as training progresses. These are implementation-specific choices, not a recommendation that every task use the same fraction, and original MBPO does not prescribe shrinking the synthetic share as real data accumulates.