Differentiable Planning › Limitations & Impact

Differentiable planning can be model-evaluation-efficient on smooth, well-conditioned problems because one forward/reverse evaluation supplies sensitivities for every action coordinate. The need for usable derivatives is structural, as is the local nature of derivative information for a pure local gradient planner. Hybrid or multistart methods can still explore multiple basins. Intrinsic conditioning depends on the model, objective, transcription, scaling, and parameterization; the solver determines how that conditioning is handled. Exploitation and memory cost depend on the broader implementation. Together with latency, safety, and accuracy requirements, these factors help determine where the method belongs.

The first limitation is locality. On a smooth objective and with a suitable step rule, gradient descent seeks a stationary point; it need not reach a local minimum or the global solution. A local derivative cannot certify which contact-mode basin is globally best, and a nearly flat maze objective may provide too little signal to discover a route around a corner. Sampling can explore more broadly when its proposal retains support over distant positive-measure regions, but finite budgets and proposal collapse prevent any automatic global guarantee. Hybrid planners can use sampling for basin discovery and gradients for local refinement. The next chapter, Search and Belief-Space Planning, turns to tree search and planning under state uncertainty.

The second limitation is differentiability itself. Hard contact-mode changes can produce discontinuous or invalid local sensitivities. Hard discrete choices provide no informative ordinary pathwise gradient for changing the selected outcome: the Jacobian of a typical continuous-logit-to-hard-choice map is zero almost everywhere and is undefined at switching boundaries. Raw symbolic states likewise need an encoding or another treatment before ordinary derivatives apply. Every decision-relevant hard component on the differentiated action-to-objective path needs a usable sensitivity route—ordinary, event-aware, implicit, surrogate, or stochastic—or it must be bypassed or optimized separately. Finite-temperature categorical relaxations and straight-through rules generally bias gradients for the original discrete problem, while score-function estimators can have high variance. Differentiable contact formulations provide useful local or surrogate gradients under their own assumptions, so contact is difficult rather than categorically impossible.

The third limitation is exploitation. At a nonstationary unconstrained point, the negative gradient supplies a local direction for reducing the model-based objective; constrained updates also need a feasible descent direction. When a reachable error makes a learned cost, terminal value, or dynamics rollout look artificially favorable, the optimizer can exploit it aggressively. There is no universal efficiency ordering against random search: gradient methods can also stall before a remote error. Defenses often add conservatism or computation, yet they may improve realized performance by preventing proxy exploitation.

The fourth limitation is conditioning and horizon. Longer horizons add sensitivity factors that can vanish, explode, or become ill-conditioned, although identity and isometric dynamics can preserve them. The behavior is shaped both by the physical system and by its learned or simulated representation. A common response is to shorten the rollout and append a terminal value, alongside preconditioning, curvature-aware methods, multiple shooting, or collocation. With a learned terminal value, that choice balances rollout-model error, terminal-value error, optimization conditioning, and computation.

Finally, there is the backward-pass cost and memory footprint. A conventional uncheckpointed reverse pass stores activation information at each step. For deterministic or equivalently replayed segments, with relevant random-number and mutable state preserved, gradient checkpointing reduces activation memory through recomputation while preserving the same gradient up to ordinary numerical reproducibility. Truncated backpropagation reduces memory and compute by cutting dependencies, but generally biases the full planning gradient unless the omitted contributions are zero. Holding per-step work, solver budget, replanning frequency, and checkpoint policy fixed, shorter horizons reduce per-call rollout work and the horizon-dependent activation-memory term; they generally lower rollout latency per planning call as well. They also reduce the number of transitions over which model error can compound, although realized error need not decrease monotonically and terminal-value error may replace it.

Differentiable models, simulators, and optimal-control layers have enabled particular model-based controllers to be optimized end to end with gradients. PILCO analytically differentiated an approximate multi-step probabilistic rollout of a Gaussian-process dynamics model with respect to policy parameters and demonstrated data-efficient robot learning. DiffTaichi-built simulators and the published Brax engine exposed gradients through supported simulation paths. MJX-JAX roughly supports gradients through its supported features, and supported differentiable Warp kernels can also provide them. These paths enable local optimization of controls and selected continuous parameters, not arbitrary structural design choices. With suitable smooth or contact-implicit formulations, such systems can optimize contact-rich trajectories. If the planner must choose an explicit contact-mode sequence, that choice needs hybrid search, enumeration, mixed-integer machinery, or an equivalent relaxation; a fixed supplied sequence can be optimized within its continuous modes. DDP and iLQR are established methods for smooth robot trajectory optimization and MPC. Current Brax guidance recommends MJX or MuJoCo Warp rather than the original Brax physics pipelines for simulation.

Modern model-based reinforcement learning uses related pieces in distinct ways. Original Dreamer used analytic pathwise gradients through imagined latent dynamics for behavior learning; later variants still learn from imagined rollouts but do not all use the same actor-gradient estimator. The planned Dreamer Family chapter will cover latent imagination and actor-critic learning across the variants. TD-MPC combines a learned latent model, reward and value predictions, and sampling-based trajectory optimization; its action search does not backpropagate the objective into sampled actions. MuZero uses a learned value at search-tree leaves rather than differentiating through tree-search decisions. A differentiable model can therefore support representation or value learning even when online action selection uses sampling or search.

The broader lesson is that differentiability is a design property of a world model. A model that predicts well but has badly scaled planner-relevant Jacobians can be a poor gradient-planning substrate. Architecture and training objectives influence that geometry, while the planning horizon changes the accumulated sensitivities. The solver, parameterization, preconditioning, and regularization determine how that geometry is handled and therefore affect its practical usefulness.