PILCO, PETS, Ensembles, and MBPO › Introduction

Imagine you are handed a physical system you have never touched before: a remote-controlled car on a smooth floor, a robot arm with unfamiliar payload, or a chemical reactor with unknown gains. You are allowed a small budget of trials, perhaps fifty or a few hundred interactions, and then judged on how well you can steer the system to a target. When that budget is small relative to the task and failures cost time, hardware, or safety, direct trial-and-error learning may be impractical. One strategy is to learn a model of how the system responds, then use that model to reason about the outcome of candidate actions before committing to any of them.

This chapter compares three decision-centric model-based reinforcement learning algorithms: PILCO, PETS, and MBPO. It also covers deep ensembles, a general predictive-uncertainty method adopted by PETS and related systems. They form a useful conceptual progression for studying model error, but not a literal inheritance chain. Deep ensembles predate PETS, PETS combines them with neural dynamics and sampled planning, and MBPO reuses a related ensemble model for policy learning rather than PETS-style online planning.

The central tension is straightforward to state and difficult to resolve. A model learned from limited data can be wrong in ways that matter. If a policy or action sequence is optimized aggressively against that model, the optimizer can select behavior that benefits from its errors, especially outside the observed data distribution. This is called model exploitation, and it explains why low average one-step prediction error need not produce good control. The methods in this chapter offer different, imperfect ways to limit that failure.

• PILCO uses Gaussian-process dynamics, analytic moment matching with repeated Gaussian approximation, and analytic policy gradients to optimize a parameterized feedback policy under a predictive state distribution.
• PETS combines probabilistic neural ensembles, particle-based uncertainty propagation, CEM, and receding-horizon control.
• Deep ensembles provide a scalable empirical proxy for epistemic uncertainty; they are useful in PETS but are a separate method with their own history and limitations.
• MBPO uses short model-generated branches from states in a real replay buffer to supply training data for an off-policy policy learner.

Before diving in, recall the frame from Part II: Inference, Decisions, and Control: a world model predicts consequences, while a decision procedure consumes those predictions. This chapter is about the interface between the two. We will distinguish what each method predicts, what assumptions it makes, and whether the model is used for online planning or for policy training. We will also lean on the treatment of uncertainty from Part IV: Space, Physics, and Causality and on model predictive control from Part VII: Planning and Agency.

Three notions of model quality recur throughout the chapter. Predictive fidelity concerns the accuracy of predictive distributions on a relevant evaluation or deployment distribution. Uncertainty quality concerns calibration, sharpness, and behavior under relevant distribution shift. Decision usefulness concerns whether the composed model-and-decision system produces good actions. A fourth notion, representation quality, asks whether the model retains the variables needed for control rather than incidental detail. This chapter defines that fourth lens but treats it only briefly; later latent-model chapters develop it directly. The properties are not interchangeable: good average predictions need not be calibrated off distribution, and good uncertainty estimates cannot rescue a defective objective or optimizer.

There is a useful continuity with system identification, filtering, and predictive control, but the correspondence is an analogy rather than a sharp historical divide. Classical identification and MPC already address model mismatch, disturbances, uncertain parameters, validation, and both hard and soft constraints. System identification is itself model learning. PETS uses sampled MPC with a learned probabilistic dynamics model. MBPO instead generates model transitions to train a policy, typically with SAC, rather than selecting actions with online MPC. These choices affect approximation and data reuse without removing the older concerns. Even when the physical plant is stationary, the estimated model can change as evidence accumulates. Likewise, an uncertainty penalty is one optional learned-control design choice, not a defining replacement for hard constraints.