Differentiable Planning › Gradient-Based Action Optimization

Once we can compute $\nabla_a J \in \mathbb{R}^{H \times m}$, action-sequence planning is a constrained finite-dimensional optimization problem:

$$\min_{a_0, \dots, a_{H-1}} \; J(a) \quad \text{subject to} \quad a_t \in \mathcal{A} \;\; \forall t,$$

where $\mathcal{A}$ is the admissible action set. Independent limits form a box: with shared scalar limits it is $[a_{\min},a_{\max}]^m$, and in general it is $\prod_i[a_{\min,i},a_{\max,i}]$. Coupled actuator limits need non-box sets; rate constraints need sequence-coupled constraints or state augmentation; state-dependent limits need state-action or path constraints such as $a_t\in\mathcal{A}(z_t)$. With continuous decision variables, the problem is a nonlinear program when its objective or explicit constraints are nonlinear. Discrete decisions instead lead to a discrete or mixed-integer formulation. The linear-quadratic example below is a convex quadratic program.

The simplest solver is gradient descent with a fixed scalar step: $a\leftarrow a-\eta\nabla_aJ$. Adaptive methods such as Adam are convenient in implementations like the worked example below. Away from its numerical stabilizer and interventions such as clipping or weight decay, Adam's moment normalization makes updates less sensitive to a uniform positive rescaling of persistent gradients. It is not fully scale-free, and it does not remove poor conditioning. For a smooth positive-curvature local model, one global step size must be stable for the direction of largest curvature, which can force slow progress in flatter positive-curvature directions. Adaptive, accelerated, and preconditioned first-order methods may reduce that penalty but do not eliminate it in general.

Two simple approaches are especially easy to implement.

Projected gradient descent takes a step and then clips each coordinate back into the box:

$$a \leftarrow \mathrm{clip}\big(a - \eta \nabla_a J,\; a_{\min},\; a_{\max}\big).$$

This enforces the box constraint exactly. At a boundary, projection removes only the component of a proposed step that points outside the box; a later descent step that points inward can still move the coordinate. Repeated outward steps can pin a coordinate at the boundary, while changing or noisy directions can make iterates oscillate along active faces. The projection operator also introduces a kink that must be handled in convergence analysis near active constraints.

Reparameterization removes the constraint entirely. Write

$$a_t = \frac{a_{\min} + a_{\max}}{2} + \frac{a_{\max} - a_{\min}}{2} \tanh(u_t)$$

and optimize the unconstrained variable $u$. In exact real arithmetic, every finite $u$ maps into the strict interior of the box. In floating-point arithmetic, $\tanh(u)$ can round to $\pm1$, and midpoint or half-width arithmetic can cancel, overflow, or underflow. The affine formula alone therefore does not universally guarantee even closed-box feasibility. With well-scaled bounds and overflow-free arithmetic, it typically lands in the closed box; apply a final clip when exact numerical box feasibility is required. The other cost is saturation: when $|u_t|$ is large, $\mathrm{d}a_t/\mathrm{d}u_t$ becomes exponentially small. The action lies very near a boundary and may become effectively stuck in finite precision, which is harmless only while the desired action remains there and harmful when an inward recovery is needed. A logistic parameterization has the same limitation.

Both methods are used in simple first-order planners. Reparameterization guarantees strict-interior feasibility in exact arithmetic but can compound vanishing gradients when it saturates. In finite precision, overflow-safe bound arithmetic followed by a final clamp, or another interval-preserving implementation, is needed to guarantee a result in the closed box. Projection can represent an exact boundary optimum directly, whereas real-arithmetic $\tanh$ approaches a boundary only asymptotically. Overall convergence still depends on scaling, step size, curvature, and active constraints.

The trajectory transcription changes both sparsity and conditioning. A key distinction is which trajectory quantities become free variables and how the dynamics are enforced.

• Single shooting optimizes only the actions and eliminates states by forward simulation. Each action-gradient block contains direct terms plus sums weighted by products of later transition Jacobians. This is the action-only formulation established above.
• Multiple shooting also promotes segment-boundary states to decision variables and adds continuity constraints. This exposes sparse local continuity Jacobians to a constrained solver and often avoids the worst full-horizon sensitivity products, although the resulting KKT system can still be ill-conditioned. It is a standard option for difficult trajectory-optimization and nonlinear-MPC problems.
• Direct collocation represents states and controls at mesh or collocation nodes and enforces integration defects. A-stable or L-stable implicit collocation schemes, such as appropriate Radau formulas, can be more stable for stiff modes than explicit integration at the same step size, although their nonlinear constraint Jacobians can still be poorly conditioned.

For sufficiently short, well-conditioned receding-horizon problems with modest path constraints, single shooting is a practical baseline with less transcription bookkeeping. As horizon-induced sensitivity products become problematic, multiple shooting can justify its additional variables and constraints; the crossover depends on dynamics, time step, dimensions, constraints, scaling, and solver. Its local continuity blocks shorten sensitivity chains but are not guaranteed to be well-conditioned.

Gradient descent uses only first-order information, and planning objectives can be badly conditioned when long-horizon sensitivities create strongly unequal curvature across action directions. Curvature-aware methods attack that local geometry directly.

Differential Dynamic Programming (DDP) and its Gauss-Newton variant iLQR compute a local quadratic approximation of the cost-to-go. Around a nominal pair $(\bar z_t,\bar a_t)$, define $\delta z=z-\bar z_t$ and $\delta a=a-\bar a_t$; let $q_z,q_a$ denote the corresponding gradient blocks of the local action cost-to-go $\mathcal Q_t$, and let $Q_{zz},Q_{za},Q_{az},Q_{aa}$ denote its Hessian blocks. When the unconstrained local control subproblem is well posed, typically after regularization makes $Q_{aa}$ positive definite, the backward algebra solves that local model exactly; it only approximates the original nonlinear problem. The backward pass augments the first-order adjoint, or value gradient, with a value Hessian:

$$\mathcal Q_t(z, a) \approx \text{const} + \tfrac{1}{2}\begin{bmatrix} \delta z \\ \delta a \end{bmatrix}^\top \begin{bmatrix} Q_{zz} & Q_{za} \\ Q_{az} & Q_{aa} \end{bmatrix} \begin{bmatrix} \delta z \\ \delta a \end{bmatrix} + \begin{bmatrix} q_z \\ q_a \end{bmatrix}^\top \begin{bmatrix} \delta z \\ \delta a \end{bmatrix},$$

and, when $Q_{aa}$ is positive definite, the local feedforward perturbation is $\delta a^\star=-Q_{aa}^{-1}q_a$. At each time step the sweep forms $Q_{aa}$, $Q_{az}$, and the local linear terms from the stage-cost derivatives, $F_t$, $G_t$, and the next value gradient and Hessian, then propagates the resulting value gradient and Hessian. Full DDP also retains contractions with second derivatives of the dynamics, while iLQR drops those dynamics terms but keeps the first and second cost derivatives. The resulting finite-horizon, generally time-varying Riccati recursion is related to, but not identical to, the discrete algebraic Riccati equation used for the infinite-horizon linear time-invariant example below.

On smooth problems with a useful local quadratic model, DDP or iLQR can converge in fewer iterations than first-order descent. The advantage is not guaranteed, especially when $Q_{aa}$ is indefinite or poorly stabilized rollouts make the local model inaccurate. DDP retains dynamics second derivatives, while iLQR drops them; both typically need regularization and a line search or trust region. With a descent-producing regularization, backtracking can reject an aggressive step until an acceptance condition is met. It can still stall or fail when no useful local direction exists.

In a smoothly evolving MPC loop, consecutive problems share structure and usually differ modestly unless disturbances, references, constraints, or modes change abruptly. Under those conditions, shifting the unused action suffix left, appending a terminal guess, and checking or restoring feasibility for the new state and constraints often gives a useful full-horizon initial sequence. The appended value can be the previous last action or zero only when it satisfies the new box, rate, coupled, and state-dependent constraints. This warm start alone does not guarantee recursive feasibility; that requires additional MPC assumptions. Warm starts can reduce the iterations needed per update, but whether this reaches a real-time deadline depends on the model, hardware, control rate, and accuracy target. The same shift-and-pad pattern is a common warm-starting discipline for CEM-based MPC, as introduced in Sampling-Based Planning and Model Predictive Control.

A useful hybrid is to evaluate a population of CEM candidates for a few distribution-update rounds to find a promising basin, then refine the best sequence with gradients. The sampling phase explores without requiring differentiability; the gradient phase performs local refinement. The outcome still depends on population size, initialization, and the objective's basin structure.

There is a second, distinct meaning of "differentiable planning" to separate from the first.

So far we have used gradients to perform the planning. We can also make the planning procedure a differentiable layer so gradients flow into upstream cost or dynamics parameters. One route solves the optimization problem in the forward pass and differentiates its KKT conditions implicitly. A standard sufficient regularity regime for classical implicit differentiation uses a differentiable local KKT residual, a strongly regular or locally unique solution branch, and a nonsingular linearized optimality system. A locally stable active set commonly permits a fixed-active-set derivative only when the problem data are differentiable and the corresponding reduced KKT system is nonsingular, conditions typically supported by constraint qualification and suitable second-order assumptions. Active-set changes may make the solution map nondifferentiable, but they do not do so in every problem. These conditions enable the classical inverse-system sensitivity formula; they are not universal necessities for every differentiable selected optimizer map. Unrolling a fixed number of solver iterations is a different route with different memory and approximation tradeoffs.

This distinction matters in learning-to-plan systems and differentiable optimization layers. It is separate from the next part of this chapter: using differentiable values inside a planning objective. The planned Policies Learned in Imagination chapter will instead examine policy learning from model-generated rollouts.