PILCO, PETS, Ensembles, and MBPO › Limitations and impact

These methods are best read as a comparison of design choices rather than a strict sequence in which each solves its predecessor's central problem.

PILCO demonstrated striking data efficiency with GP dynamics, analytic moment matching with repeated Gaussian approximation, and analytic policy gradients, but cubic GP scaling and restrictive analytic forms limit its practical range. Its expected-cost formulation and bounded objective remain useful examples, although direct historical inheritance by every later method should not be assumed.

PETS showed sample-efficient learned control on challenging simulated continuous-control benchmarks, including a contact-rich simulated 7-DOF Pusher task and a separate 7-DOF Reacher task. The paper did not report those experiments on physical robots. PETS pays for online planning with ensemble inference, particle rollouts, and CEM search. Particle count controls return-estimation precision, while action dimension and horizon primarily make action-sequence optimization harder; those are related compute costs, not the same phenomenon.

MBPO's published analysis proposes a lower-bound tradeoff between policy divergence and current-policy model error under truncated branches. Because a later reproduction disputes pre/post-branch indexing in the proof, its printed expression is a motivation for positive model use, not a settled performance certificate. The analysis does not establish that earlier work was generally misguided or prescribe a universally optimal $k$. Its practical design introduces rollout-schedule and replay-mixture choices that require empirical validation.

Several recurring failure modes remain useful as a checklist:

• Distribution shift. Optimization can seek states poorly represented by model-training data. Coverage, regularization, constraints, explicit risk objectives, and real-state branch starts can reduce this risk without eliminating it.
• Accumulated rollout error. Imperfect open-loop learned-model predictions can diverge, saturate, oscillate, or cancel. Shorter horizons and new observations reduce consecutive reliance on them; an exact model would not drift.
• Uncertainty mis-estimation. Ensemble disagreement is an empirical proxy and can be miscalibrated. A collapsed total variance discards the within-/between-member distinction; retaining both terms permits an operational decomposition, but neither is a pure uncertainty category under misspecification.
• Objective and usage misspecification. A saturating cost and an explicit uncertainty penalty change the objective, while a short rollout limits model use. These are different mechanisms and can fail in different ways.

This list identifies common risks, not a complete safety certificate or a prediction that every controller exhibits all four. The methods advanced learned model-based RL on benchmark control problems and supplied mechanisms that recur in later work. Later latent-control methods may reuse short planning horizons, terminal values, ensembles, or uncertainty proxies in different combinations. For example, TD-MPC and Control-Centric Representations uses short latent trajectory optimization with a terminal value; that is not the same as MBPO's branched synthetic policy data. The following chapter, World Models, SimPLe, and PlaNet, compares distinct modeling choices, including compact latent dynamics and action-conditioned video prediction.