Classical Planning and Optimal Control › Summary

• Planning turns a predictive model into a controller. Given $p(s' \mid s, a)$ and a reward, the planner finds actions that maximize expected return. Open-loop plans are action sequences; closed-loop policies are state-dependent and can react to observed outcomes.
• State-space search handles finite graph problems. Dijkstra is uninformed; A* uses a heuristic and is optimal under the admissibility and graph-search conditions stated in the chapter.
• For bounded rewards and $\gamma<1$, tabular dynamic programming solves the Bellman optimality equation and value iteration converges geometrically. Stochastic transitions fit naturally, but state enumeration causes the curse of dimensionality.
• LQR exploits linear dynamics and quadratic costs to obtain $u=-Kx$ from the Riccati equation. Its stability guarantee applies to the modeled system under stabilizability and detectability conditions; certainty-equivalent deployment on an estimated plant needs a robustness check.
• Trajectory optimization computes finite-horizon state-control sequences by shooting or collocation. Numerical hard-constraint satisfaction is tolerance- and discretization-dependent, and local solvers are sensitive to initialization.
• All four methods depend on a model that is adequate in the region used for decisions. With learned models, rollout error and optimizer-induced model exploitation become first-class problems.

The next chapter develops a probabilistic framing of control. Control-as-inference methods introduce optimality or preference variables and infer a posterior over trajectories and actions. Exact inference equivalences are narrow, for example maximum-entropy formulations with suitable deterministic dynamics, while stochastic systems generally require approximations such as variational inference. Uncertainty drives epistemic exploration only when the objective includes terms such as information gain or expected free energy. Those conditions lead into control as inference and active inference.