Sampling-Based Planning and Model Predictive Control › Sampling-Based Planning and Model Predictive Control

Imagine a delivery drone waiting at the edge of a windy courtyard. Its world model is a neural network trained on logs from previous flights. The network accepts a state and an action and returns the next state, but it is a black box. With piecewise-linear activations, its input-output map may have derivative discontinuities; its gradients may also be noisy or uninformative in regions poorly covered by training data. More fundamentally, it was trained to predict, not to be optimized. A next-state prediction loss has only an indirect relationship to the outcome we care about: whether the drone reaches the landing pad. How do you use the model to pick what to do?

Learned world models often make this problem more visible than hand-derived dynamics. Some simulators expose analytic derivatives and structure that an optimizer can exploit; others are black boxes too. A differentiable learned model supplies derivatives through backpropagation, but derivatives evaluated on out-of-distribution states can be technically well-defined and still unreliable for planning. A gradient-based planner inherits that weakness.

The core insight of this chapter is that you do not need to differentiate the model to plan with it. You only need to be able to run it forward. Given a candidate sequence of controls, you can roll the model out step by step and ask, "what would happen if I did this?" Then you can compare candidate sequences by the cost of the trajectories they produce and pick the best one. This is sampling-based planning: convert control into search, and search over action sequences by simulation.

The idea is old and appears in several related forms. Evolutionary optimization searches over populations of candidate policies or plans. Shooting methods in optimal control generate trajectories by propagating candidate control sequences through the dynamics. Path-integral methods form a more specific stochastic-control family with additional assumptions and importance-weighted updates. Their shared practical trade is narrower than an equivalence: derivative-free search can query a forward model without requiring analytic gradients, but it offers no general guarantee of finding a global optimum. When model rollouts can be batched efficiently, that trade is often useful for learned dynamics.

A common way to deploy this idea is model predictive control (MPC). Instead of committing to a long plan produced once, MPC plans over a finite horizon, executes the first action or a short block, observes the result, and replans from the new state. This closes the loop between the model and the environment, allowing feedback to correct some disturbances and model mismatch. Receding-horizon control long predates learned models and is widely used in industrial process control, as documented in Qin and Badgwell's industrial survey. Learned-model MPC changes the model inside the loop while retaining the same basic feedback structure.

In this chapter we build up this family of methods from the ground floor. We start with the simplest possible planner, random shooting, which samples many action sequences uniformly and keeps the best. We then introduce the cross-entropy method (CEM), which iteratively narrows a sampling distribution around the actions that worked. From there we assemble a full model predictive controller and see how replanning can compensate for some model error and disturbance. We then look at the design decisions that decide whether an MPC stack works in practice: how long the planning horizon should be, how often to replan, how to enforce constraints, how to warm-start the optimizer from the previous step's plan, and how to fit all of this inside a real-time control budget.

Building on the optimal-control foundations from Part II, this chapter treats MPC as a practical, sampling-based realization of the receding-horizon principle. It sits at the boundary between the pure predictive objectives of Part VI and the learned-policy imagination methods we will meet in Part VII, Chapter 4 and the decision-centric research lineages of Part VIII. Everything here assumes you have a forward model, either a hand-written simulator or a learned one. The chapter is about how to wield it.

It is worth being explicit about how this material lines up with a classical model-based control pipeline. System identification estimates dynamics from input-output data; state estimation enters when the state is not directly measured; and an optimal-control method chooses actions against that model and estimate. Under linear dynamics, quadratic objectives, and convex constraints, an MPC subproblem is often a quadratic program. Learned-model variants retain the decomposition but may replace physical states with latent states and structured dynamics with flexible function approximators. These are tendencies, not a clean classical-versus-learned dichotomy: classical models can be nonlinear and high-dimensional, while learned models can be smooth and structured. The relevant solver depends on the assumptions that the particular model, objective, and constraints satisfy.

One clarification helps before we start. When we say a model is "good," we mean it is good for a specific purpose, and it is worth separating four properties that are often blurred together.

• Predictive fidelity is how accurately the model's one-step or multi-step rollout matches the true system on states and actions that the training distribution covers. This is what a supervised loss measures, and it is the property most papers report.
• Representation quality is whether the model's internal state carries the information a planner needs, even if that representation is not a physically meaningful state vector. A latent model can have excellent representation quality while its decoded states look strange or discontinuous.
• Uncertainty quality is whether the model knows what it does not know, meaning that regions of state and action space far from the training data are flagged as uncertain instead of being confidently extrapolated.
• Decision usefulness is whether the plans that optimize against the model, once executed, make progress on the task. This is the property MPC needs, and it is not implied by the other three.

A model can have high average predictive fidelity and still be poor for planning, because optimization may select action sequences in regions where the model is systematically optimistic. Conversely, a model with mediocre average fidelity but accurate task-relevant local predictions can still support useful control. Sampling-based MPC exposes this separation directly: its decisive test is closed-loop performance under the planner's induced state-action distribution, not prediction loss alone.