PILCO, PETS, Ensembles, and MBPO › Uncertainty-aware model predictive control

"Uncertainty-aware" can describe several different mechanisms. Keeping them separate prevents a risk-neutral estimator from being mistaken for a safety objective.

The first role is uncertainty propagation under an expected-return objective. PETS samples predictive trajectories and averages their returns. This estimates expected return under the fitted ensemble predictive distribution. It accounts for how uncertainty passes through nonlinear dynamics and rewards, but it imposes no generic penalty on return variance or ensemble disagreement.

The second role is explicit risk sensitivity or an uncertainty penalty. One possible objective is

$$\operatorname{score}(a_{t:t+H-1}) =\mathbb{E}[R]-\lambda\mathcal{U}(a_{t:t+H-1}),$$

where $\mathcal{U}$ might measure return spread, a lower-confidence adjustment, or another task-defined uncertainty statistic. The coefficient $\lambda\ge 0$ makes the risk tradeoff explicit and must put the uncertainty term on a scale commensurate with return. Worst-case aggregation and CVaR are other possibilities. Baseline PETS does not include such a term.

The third role is uncertainty as an information signal. A controller may choose actions partly because the resulting observations are expected to improve its model. That dual-control objective values future information rather than merely charging for present uncertainty.

A method can combine these roles, but its behavior should be attributed to the actual objective. PETS performs uncertainty propagation with risk-neutral averaging. An added penalty would be a separate modifier, and an information bonus would add an exploration motive.

For PETS, MPC executes one action from each optimized sequence and then replans from a new observation. This limits the uncorrected forecast span used by one plan and limits commitment to its first action. It does not place a numerical bound on error or make global task success depend only on local accuracy. Reported PETS horizons are on the order of tens of steps for its tasks, but the required horizon is task-specific; delayed rewards, terminal conditions, constraints, and necessary travel through poorly modeled regions can all require additional design.

Suppose ensemble members disagree near a joint limit. If that disagreement affects reward-relevant predictions within the planning horizon, particle sampling can produce a wider or differently shaped return distribution for action sequences entering that region. Under arithmetic averaging, disagreement alone has no direction: pessimistic and optimistic deviations may cancel, reward curvature may lower or raise the mean, or the mean may remain unchanged. CEM ranks the resulting means unless an explicit risk criterion is added.

Consequently, baseline PETS does not automatically avoid every blind spot, and ensemble size or regularization is not a principled caution knob. If a controller with an explicit uncertainty penalty avoids a necessary region, reducing that penalty is one possible response. If risk-neutral PETS avoids the region because its estimated expected return is poor, the relevant remedies are better targeted data, an information objective, a changed task objective, or a deliberately risk-sensitive decision rule. There is no penalty to relax in baseline PETS.

Aleatoric and epistemic uncertainty can both alter expected return through nonlinear dynamics and rewards; neither has a universal "variance only" or "downward bias" effect. Their practical distinction is instead about what can be reduced. Within-member predictive variance is an aleatoric proxy under a well-fitted member model, while between-member mean disagreement is an epistemic proxy. Retaining both terms preserves a useful operational decomposition; collapsing them into total variance loses that distinction. Even when retained, the terms do not uniquely identify pure aleatoric and epistemic uncertainty: model misspecification can affect within-member variance, and training variability or shared error can affect between-member disagreement.

Irreducible variability may call for expected-outcome, robust, or risk-sensitive planning. Reducible disagreement may motivate data collection, but only after checking that the ensemble spread is calibrated enough to serve as a local signal. Readers interested in the deeper treatment of stochasticity should revisit Part IV, Ch 5: Stochasticity and Uncertainty.

The main compute allocations have different jobs:

• Ensemble size $N$. More members can reduce finite-ensemble estimation noise and may improve functional diversity, but correlated or systematically biased members need not become a better posterior proxy. Training and inference cost grow approximately linearly before batching effects.
• Particles per candidate $P$. More particles improve predictive-distribution coverage and reduce Monte Carlo noise in each candidate's expected-return estimate. They do not increase the number of action sequences searched.
• CEM candidate population $M$. More candidates increase action-sequence search coverage within an iteration.
• Horizon $H$. A longer horizon exposes delayed consequences but uses more modeled steps, which can increase sensitivity to model error.
• CEM updates and elite fraction. More updates can refine the proposal. Too small an elite fraction can collapse it prematurely, while too large a fraction weakens each update.

These parameters share a compute budget and should be tuned jointly, but one does not mechanically justify increasing another. A larger ensemble does not prove that a longer horizon is safe, and more particles do not by themselves justify more CEM iterations.

Failure signatures should follow the same distinctions. Too few candidates underpower action search. Too few particles produce noisy or poorly resolved candidate-value estimates. Too few or highly correlated ensemble members make disagreement unreliable. An excessive horizon exposes the score to longer model rollouts. Aggressive elite selection can lock onto sampling noise. CEM can fail abruptly through premature collapse or failure to sample a narrow optimum, so graceful degradation should not be assumed.