Differentiable Planning › Differentiable Planning

In Sampling-Based Planning and Model Predictive Control, we planned by proposing action sequences, rolling each one through the world model, scoring the result, and keeping the best. Derivative-free population methods remain usable when model gradients are unavailable or unreliable, but their performance depends on the proposal, objective, and evaluation budget. Random shooting with a twenty-step horizon and six action dimensions searches a 120-dimensional space. If each added coordinate has conditional probability at most a fixed $q<1$ of landing in its acceptable subset under the proposal, then one sample's success probability is at most $q^d$ after $d$ such coordinates. Independent uniform coordinates with a fixed acceptable range fraction $q$ are one example. This is the relevant curse of dimensionality, not a guarantee that every high-dimensional planning problem is hard.

The derivative-free random-shooting planner used in the previous chapter does not compute or use an action-space derivative telling it which infinitesimal change lowers cost. Random shooting relies on sampled candidates, while CEM uses elite-sample statistics to concentrate its proposal on regions found promising. If the proposal assigns probability $p$ to near-good plans, $N$ independent candidates contain at least one with probability $1-(1-p)^N$. When adding coordinates makes $p$ shrink exponentially, maintaining the same success probability requires exponentially more candidates. On smooth problems with reliable gradients, ignoring those derivatives can therefore require many more model evaluations to locate a useful update. The advantage is conditional: unstable, biased, or locally misleading gradients can erase it.

Differentiable planning asks a different question. Instead of "which of these candidates is best?", it asks "in which local direction should I change this candidate?" If the rollout and cost are differentiable along the executed trajectory, reverse-mode automatic differentiation gives the entire action-gradient vector at a small constant multiple of the cost of one objective evaluation, independent of the number of action coordinates. For cost minimization, whenever $\nabla_a J\neq0$, the negative gradient is a local descent direction. With a suitable step size and constraint handling, any nonzero feasible negative-gradient or projected-gradient direction reduces cost locally and updates the action blocks with nonzero components. This direct sensitivity calculation can replace or complement population sampling.

The recipe is short enough to fit in a paragraph. Start with a guess for the action sequence. Roll the model forward from the current state, recording every intermediate state. Evaluate the cost of that trajectory. Backpropagate the cost through the rollout to obtain $\partial J / \partial a_t$ for every time step $t$. Take a gradient step on the action sequence. Repeat until a declared stopping rule or computation budget is met, such as a maximum iteration count paired with an objective-improvement tolerance and patience window. Then apply the first action and replan from the new state. Planning has become optimization.

This reframing has strict prerequisites and important failure modes:

• The action-to-objective computation must provide a usable action derivative or derivative estimator. Ordinary reverse mode needs usable local pullbacks along the executed path. Many neural latent-dynamics models for continuous control use differentiable operations on this path, while stochastic objectives can instead use a valid derivative estimator. Hard contact and friction switches can introduce kinks or jumps. Discrete mode changes can also produce branch-dependent derivatives.
• Decision-relevant objective terms on the differentiated action-to-objective path must contribute a usable action derivative or valid derivative estimator; separately handled terms need not. Quadratic tracking costs are differentiable, although their action signal can vanish or be poorly scaled. A hard indicator cost ("one if you miss the target") has a zero, and therefore uninformative, pathwise gradient away from its boundary; at the boundary it is generally nondifferentiable. A smooth surrogate can supply a pathwise gradient. For an expected cost under a suitable stochastic formulation, a score-function estimator can be valid under the regularity conditions developed later, though it often has high variance and may need variance reduction; if it is not usable, choose another valid sensitivity route. If differentiation through the hard term is bypassed, preserve its decision effect through a constraint, full-objective candidate evaluation, or an outer discrete or hybrid search.
• The gradients must remain numerically usable across the planning horizon. A long rollout contains long Jacobian products. Those products can shrink, grow, remain bounded, oscillate, or show transient amplification; strong contraction, expansion, and ill-conditioning are the failure modes that concern us here.

Each condition requires its own tools. This chapter builds the derivative machinery, examines how costs and terminal values shape the objective, and then focuses on numerical gradient usability, a practical failure mode over long horizons.

We derive backpropagation through dynamics as an adjoint recursion, translate it into action optimization, examine what makes a cost or value function usable as a planning objective, and confront the numerical realities of long rollout derivatives. Along the way we build a complete differentiable planner for a double integrator and then isolate its gradient chain under contracting and expanding scalar dynamics.