Differentiable Planning › Summary

• Differentiability makes action-sequence planning amenable to gradient optimization. If $f$ is differentiable in state and action and the stage and terminal costs are differentiable in their relevant arguments along the rollout, reverse mode returns the full action gradient at a small constant multiple of one objective evaluation. This removes the coordinate-wise finite-difference factor and can help in high-dimensional continuous action spaces. Naive random shooting suffers exponentially only when good plans occupy a shrinking fraction of its proposal distribution.

• The backward pass is a discrete adjoint recursion. With $\lambda_t=\partial J/\partial z_t$, the recursion is $\lambda_H=\partial c_H/\partial z_H$ and $\lambda_t=\partial c_t/\partial z_t+F_t^\top\lambda_{t+1}$; the action gradient is $\partial J/\partial a_t=\partial c_t/\partial a_t+G_t^\top\lambda_{t+1}$. This is BPTT for the discrete recurrence and the discrete-time analogue, not the identical equation, of a continuous ODE adjoint.

• The gradient is a sum of Jacobian products. Unrolling the recursion shows terms containing $\prod F_j^\top$. These products are a major source of poor conditioning in single shooting. Action Jacobians can improve or worsen conditioning depending on their rank, scale, and alignment. Cost curvature can worsen it when scales are highly unequal, while well-scaled positive control curvature can regularize the action Hessian. Active or poorly scaled condensed constraints can degrade the constrained Jacobian or KKT system. Cancellation among future-cost contributions is a separate problem that can weaken the net gradient signal. A poorly scaled or globally coupled trajectory parameterization can worsen conditioning, while a good one can act as preconditioning.

• Bounds and conditioning require explicit choices. Projection represents exact box boundaries but can pin coordinates. In exact real arithmetic, finite $u$ in a $\tanh$ parameterization gives strict-interior feasibility. Finite-precision $\tanh$ may round to $\pm1$; overflow-safe bound arithmetic with a final clamp, or another interval-preserving implementation, is needed to guarantee closed-box feasibility numerically. Adam reduces sensitivity to persistent coordinate-wise scaling without fixing poor conditioning. Regularized DDP or iLQR can mitigate curvature anisotropy; on nonlinear or non-convex problems, they typically also need a useful local model and globalization such as a line search or trust region. Warm starts often reduce MPC iterations; real-time sufficiency remains system- and hardware-dependent.

• Decision-relevant objective terms on the differentiated path need a usable action derivative or derivative estimator. Separately handled terms need not. Smooth quadratic costs have analytic derivatives, but they provide a useful action-space signal only where the composite rollout derivative has usable direction and scale. Smooth sparse-cost surrogates built from autodiff-supported operations, such as the softplus and Gaussian examples above, are straightforward to differentiate. They provide a useful local planning signal only where their weighted gradients have usable direction and scale relative to other terms, estimator noise, numerical precision, and the optimizer. For differentiable inequalities and strictly feasible trajectories, log barriers are differentiable in the interior and grow without bound as the constraint boundary is approached. When learned costs or values are differentiated directly by ordinary pathwise reverse mode, their planner-relevant computations need suitable differentiable operations; otherwise they need another valid sensitivity estimator or must be handled separately. They also need decision fidelity on planner-relevant states, because reachable approximation errors can be exploited. Differentiability is not inherently an error source, although a smooth surrogate or restricted differentiable function class can introduce bias when the target is nonsmooth or discontinuous.

• Long-rollout sensitivity depends on products of state Jacobians. For a constant scalar Jacobian $F=\rho$, an $L$-factor sensitivity product has magnitude $|\rho|^L$. For a constant normal Jacobian, each excited eigenmode scales as $|\lambda_i|^L$, and its Euclidean induced operator norm (spectral $2$-norm) is $\lVert F^L\rVert_2=\max_i|\lambda_i|^L$. General finite-horizon behavior depends on singular-value growth along propagated directions, including non-normal transient growth. Shortening the horizon and reformulating the trajectory problem can mitigate poor products. Loss scaling and higher precision can address some magnitude, range, and roundoff consequences without changing the exact physical sensitivity map. Consistent state and action rescaling can additionally improve the numerical representation and conditioning of that map, but it does not remove genuine physical contraction, expansion, rank loss, or mode discontinuities. Well-chosen preconditioning or suitably regularized curvature-aware methods can improve how the optimizer handles the resulting conditioning when their transformed or local model is accurate enough; poor choices need not help. Decision-relevant hard components that block a usable derivative require a valid fixed-branch or event-aware sensitivity, a relaxation or estimator, or a hybrid bypass or separate discrete search.

• A short horizon plus a suitable terminal value can replace part of a long rollout. This reduces the transition-chain length but trades omitted-tail error for terminal approximation error. What matters is preserving decision-relevant values or gradients over candidate terminal states, not absolute value accuracy everywhere.

• One-step prediction accuracy and closed-loop usefulness are different properties. A model can score well on prediction and still produce poor plans if its local sensitivities are inaccurate or badly scaled. Architecture, data, objectives, derivative regularization, and the planner together determine whether multi-step gradients are useful.

The next chapter, Search and Belief-Space Planning, turns from a single smooth action sequence to tree search and planning under uncertain state. That shift preserves the objective-design lessons here while changing how the planner represents alternatives and uncertainty.