Classical Planning and Optimal Control › Limitations and Impact
World Models HandbookAll chapters Classical Planning and Optimal Control › Limitations and ImpactOne section · Sections in this chapter8 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Many assumptions behind these methods motivate research questions later in the book.
The deepest shared assumption is that the model is adequate where the planner uses it. Once the model is learned, two new failure modes appear. First is compounding rollout error. Small one-step errors are propagated by the dynamics and can accumulate, cancel, or amplify depending on their bias, correlation, and the system's sensitivity. A systematic error of size can contribute a sum of propagated errors over an -step open-loop rollout. In feedback, corrupted states also change later actions. One-step accuracy therefore does not determine multi-step fidelity, so both should be evaluated at the horizons used for decisions.
The compounding mechanism can be illustrated with the double integrator itself.
import numpy as np
## Seeded Monte Carlo illustration on the double integrator. The two error
## models have matched one-step RMS magnitude: one is zero-mean independent
## noise, while the other has a fixed velocity bias.
dt_e = 0.1
A_true_e = np.array([[1.0, dt_e], [0.0, 1.0]])
x0_e = np.array([0.0, 1.0])
sigma_e = 0.02
n_rollouts = 2000
bias_step = np.array([0.0, np.sqrt(2.0) * sigma_e])
rng_e = np.random.default_rng(17)
horizons = np.arange(1, 51)
sq_err_unbiased = np.zeros((n_rollouts, len(horizons)))
sq_err_biased = np.zeros(len(horizons))
x_true = x0_e.copy()
unbiased_states = np.repeat(x0_e[None, :], n_rollouts, axis=0)
biased_state = x0_e.copy()
for i, H in enumerate(horizons):
x_true = A_true_e @ x_true
unbiased_states = unbiased_states @ A_true_e.T
unbiased_states += rng_e.normal(0.0, sigma_e, size=unbiased_states.shape)
biased_state = A_true_e @ biased_state + bias_step
sq_err_unbiased[:, i] = np.sum((unbiased_states - x_true) ** 2, axis=1)
sq_err_biased[i] = np.sum((biased_state - x_true) ** 2)
err_unbiased = np.sqrt(sq_err_unbiased.mean(axis=0))
err_biased = np.sqrt(sq_err_biased)
The injected one-step errors have matched RMS magnitude, yet their rollout errors differ markedly. Independent zero-mean disturbances still accumulate in this marginally stable system, while the fixed velocity bias compounds much faster. This controlled example does not establish a universal growth law; it shows why the temporal structure of model error matters and why rollout fidelity should be measured directly.
Second is model exploitation. Optimization can steer a planner toward inputs where an approximate model is optimistically wrong, especially outside the data distribution. Approximation creates this risk but does not imply that every model will be exploitable on every task. Classical optimizers do not remove the problem when they plan with an approximate model. Practical defenses include uncertainty-aware or robust objectives, conservative penalties, shorter horizons, feedback and replanning, and collecting new data in planner-visited regions. Chapter 6 develops this gap between predictive fidelity and decision usefulness directly.
The second limitation is scale. Cartesian tabular representations grow exponentially in the number of factored dimensions at fixed resolution, although a particular search problem may explore only a small reachable subset. LQR remains efficient by restricting dynamics and costs to the linear-quadratic class. Nonlinear trajectory optimization with a local solver is initialization-sensitive and may reach a local stationary point, an infeasible point, or fail; convex special cases have stronger global guarantees. Discretizing a continuous space exposes a familiar trade-off: fine grids improve resolution but grow combinatorially, while coarse grids may erase task-relevant structure. Part III turns to representations that aim to make approximate planning tractable.
A third limitation is that these planners typically assume full observability and a stationary, Markov environment. We know from Chapter 1 of this part how to restore Markovianity through belief states, and from Chapter 2 how to propagate those beliefs forward. But belief-augmented planning (search over beliefs, DP over beliefs) multiplies the state space again, since a belief is a distribution over states and the space of distributions is vastly larger than the space of states, and making that tractable is an open engineering frontier that connects directly to the memory architectures of Part III.
Under their stated assumptions, these methods do offer strong guarantees: Dijkstra and appropriately configured A* return optimal graph paths; discounted finite-MDP value iteration converges; and infinite-horizon LQR stabilizes its modeled plant under stabilizability and detectability conditions, with a quadratic certificate under the positivity conditions stated above. Learned controllers can also be analyzed or certified in restricted settings, but classical baselines make the assumptions unusually explicit. Hybrid systems often use learning to supply a model or representation and classical optimization to choose actions; how dominant that pattern is depends on the application domain.
The conceptual takeaway is the division of labor: the model predicts and the planner chooses. Search, DP, LQR, and trajectory optimization implement that split differently, trading off model class, planning horizon, computation, and available guarantees. Many later agents can be analyzed in the same trade space even when model and planner are implemented jointly.