Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Classical Planning and Optimal Control › From Prediction to Decision

Michael BrenndoerferJune 10, 20264 min read

World Models HandbookAll chapters

Part 2 of 12Inference, Decisions, and Control

Chapter 10Classical Planning and Optimal Control › From Prediction to Decision

Reading section 2 of 10 · From Prediction to Decision · 4 minSection 2 of 10 · 4 min

Section 2 of 10 · 4 min read

Classical Planning and Optimal Control › From Prediction to DecisionOne section · Sections in this chapter2 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's separate the two roles cleanly, because blurring them causes endless confusion.

The transition model answers a predictive question: given a state and an action, what happens next? In notation we have used throughout Part II, it provides $p(s_{t+1} \mid s_t, a_t)$ . A learned reward model may also predict the numeric value of an externally defined objective. The transition dynamics are descriptive; the reward or cost supplies the normative criterion used by the planner. A perfect weather simulator predicts well but does not, by itself, specify what an agent should prefer.

The controller or planner answers a normative question: given that predictive machinery, which actions should we take to accumulate the most reward (or the least cost) over some horizon? The planner is prescriptive. It consumes predictions and produces commitments. The two components can be developed and unit-tested separately, but they must be validated jointly because planning changes which parts of the model matter.

The planning problem

Given an initial state $s_0$ , a model of the environment, a reward or cost function, and a planning horizon, find an action sequence or a policy that maximizes expected cumulative reward. For an open-loop sequence, we seek

\max_{a_0, a_1, \dots, a_{T-1}} \; \mathbb{E}\left[\sum_{t=0}^{T-1} \gamma^t r(s_t, a_t) \right],

where:

$a_0, a_1, \dots, a_{T-1}$ : the open-loop action sequence being optimized, one action per time step
$T$ : the planning horizon, the number of steps into the future
$r(s_t, a_t)$ : the reward assigned by the planning objective to state-action pair $(s_t,a_t)$ , possibly supplied by a separate learned reward model
$\gamma$ : the discount factor, weighing future rewards against immediate ones
$\mathbb{E}[\cdot]$ : the expectation over successor-state randomness under trajectories consistent with the planning transition model; the reward or cost may be specified separately from that model.

The formulation above hides an important choice. An open-loop plan commits to an action sequence before seeing where the actions land, while a closed-loop policy can react to new state measurements or estimates. The basic search and trajectory-optimization forms used here produce open-loop plans; tabular dynamic programming and LQR produce feedback policies. Receding-horizon methods partially close the loop by replanning from new information, which can limit error accumulation but does not guarantee robustness to a bad model.

Two distinctions from earlier chapters matter immediately. First, open-loop prediction quality and closed-loop decision usefulness can diverge. A model can fit one-step transitions in a validation set yet remain biased along the directions a planner exploits. Conversely, a controller can tolerate sizable task-irrelevant or feedback-correctable errors when the resulting closed loop remains stable. Predictive fidelity, representation quality, uncertainty quality, and decision usefulness therefore need separate tests. Second, planning from a model means reasoning about futures that have not been experienced, which is valuable precisely where multi-step model error becomes dangerous.

A useful mental split for this whole chapter:

Search asks: what is the best single path from here to there?
Dynamic programming asks: what is the best action from every state?
LQR asks: what is the optimal feedback law for known linear dynamics and quadratic cost?
Trajectory optimization asks: what state-and-control trajectory best satisfies the dynamics, objective, and any task constraints?

These four questions sit on a spectrum of scope. The A* example answers one start-goal query. Tabular dynamic programming amortizes work over every represented state. LQR gives a feedback formula under linear-quadratic assumptions. Nonlinear trajectory optimization accepts broader dynamics and costs, but still relies on a finite parameterization and on solver assumptions such as differentiability or constraint regularity.

We now take these in turn, using one running example, a small grid robot, for the discrete methods, and a double integrator plus a pendulum for the continuous ones.