Differentiable Planning › Differentiable Value and Cost Functions

The planning objective is a consequential design choice because it defines what the optimizer is asked to prefer. With accurate gradients and a suitable step-size or line-search rule, a gradient-based planner follows local descent directions in that objective, even if it has a convenient minimum in the wrong place.

A standard stage cost for smooth continuous control and trajectory tracking is

$$c_t(z_t, a_t) = (z_t - z^\star)^\top Q (z_t - z^\star) + a_t^\top R a_t,$$

with $z^\star$ the target state, $Q\succeq0$ weighting state error, and $R\succ0$ penalizing effort. For physical coordinates with documented units and suitably structured, calibrated $Q$ and $R$, the terms can be interpreted as weighted state-error and action-effort contributions. For a learned latent state, the first term is a $Q$-weighted squared discrepancy—a squared seminorm if $Q$ is singular—and need not align with task semantics. Without a bound on decoder sensitivity, numerically close latents can still decode to substantially different observations. For a decoder $d(z_t)$ and target $x^\star$ in $\mathbb R^p$, a decoded cost $(d(z_t)-x^\star)^\top Q_x(d(z_t)-x^\star)$ with $Q_x\succeq0$ in $\mathbb R^{p\times p}$ measures error in observation coordinates and costs a decoder evaluation per step. It is reliable as a task-semantic cost only if the observation metric is appropriate and decoder error over planned states does not materially distort task-relevant costs, rankings, or gradients. See Sufficient State and State Abstraction for the representation question.

Other common forms:

• Smooth surrogates for sparse costs. Away from its boundary, a binary goal term has zero pathwise gradient and gives an action optimizer no local signal from that term. For $\beta>0$, a smooth alternative is $c(z)=\beta^{-1}\log\!\left(1+\exp\left(\beta(\lVert z-z^\star\rVert^2-\epsilon^2)\right)\right)$, a softplus approximation to the hinge violation $\max(0,\lVert z-z^\star\rVert^2-\epsilon^2)$ rather than to a binary indicator. A Gaussian goal bonus, $c(z)=-\exp(-\lVert z-z^\star\rVert^2/2\sigma^2)$, is another option. At fixed distance $r=\lVert z-z^\star\rVert>0$, its state-gradient magnitude is $r\sigma^{-2}\exp(-r^2/2\sigma^2)$ and is largest at $\sigma=r/\sqrt{2}$. In the tail $r>\sqrt{2}\sigma$, shrinking $\sigma$ further weakens the gradient; for $\sigma>r/\sqrt{2}$, shrinking it can strengthen the gradient. Tune $\sigma$ to the candidate-distance distribution, and narrow it as candidates approach only while preserving a useful signal.
• Logarithmic barriers. For a barrier weight $\kappa>0$, an inequality $g(z)>0$ on a strictly feasible trajectory can be represented by $c(z)=-\kappa\log g(z)$. The term inherits $g$'s regularity in the strict interior: it is smooth there when $g$ is smooth, and merely differentiable there when $g$ is merely differentiable. It diverges to $+\infty$ as $g(z)\to0^+$. At any finite positive $\kappa$ it is an interior approximation, not an exact replacement for a general hard constraint, and it cannot be evaluated at an infeasible point.
• Penalty terms. For a penalty weight $\rho>0$, softer alternatives add $\rho \max(0, -g(z))^2$. Penalties allow small violations in exchange for lower cost elsewhere; barriers do not.
• Learned costs. A neural cost can be learned from demonstrations through inverse optimal control or from explicit preferences through reward modeling. Smooth operations make it differentiable everywhere; standard ReLU networks provide gradients almost everywhere. Approximation and generalization errors, especially off distribution, can become exploitable when they make reachable plans look artificially attractive.
• Energy-based and contrastive costs. If a model provides a differentiable energy whose low-energy configurations align with task success, that energy can serve as a planning cost. Predictive compatibility alone does not guarantee control utility; see the chapter on predictive embeddings and energy-based models.

Cost shaping requires caution. A generic dense auxiliary term can change the optimal plan. For this finite-horizon, undiscounted cost convention, one invariant construction is

$$c'_t=c_t+\Phi_t(z_t)-\Phi_{t+1}(z_{t+1}),\qquad c'_H=c_H+\Phi_H(z_H).$$

The added terms telescope to the fixed constant $\Phi_0(z_0)$, so they do not change the ordering of feasible action sequences. Omitting the terminal correction leaves the residual $-\Phi_H(z_H)$, which is generally action-dependent and can change the optimum when $\Phi_H(z_H(a))$ varies across feasible plans. Discounted reward shaping uses a different sign and includes the discount factor; the reinforcement-learning result does not transfer verbatim without matching those conventions.

When a finite horizon approximates a longer or infinite-horizon objective, it omits a tail whose value can vary across terminal states. A terminal cost can approximate that continuation:

$$J_H^V(a) = \sum_{t=0}^{H-1} c_t(z_t, a_t) + V(z_H), \qquad V(z) \approx \inf_{\pi} \; \mathbb{E}_{\pi}\Big[\sum_{k=0}^{\infty} c(z_k, a_k) \;\Big|\; z_0 = z\Big],$$

assuming the continuation cost is finite and uses the same dynamics, constraints, costs, and information pattern as the planner. If $V$ is the exact optimal continuation cost, the subproblems attain their minima, and $H\geq1$, dynamic programming makes every minimizing prefix begin with an action in the same infinite-horizon optimal first-action set. Without a unique optimum or consistent tie-breaking, different horizons may select different actions from that set. In MPC, suitable terminal costs—often together with terminal sets and other conditions—can support stability arguments. Structurally, the terminal value is the planning analogue of an $n$-step temporal-difference bootstrap: both replace an omitted tail with a value estimate, but a planning terminal value guides action optimization only when it distinguishes reachable terminal states, while the bootstrap supplies a learning target.

For an equilibrium-centered, unconstrained infinite-horizon LTI linear-quadratic regulator, under the usual stabilizability and detectability assumptions, the value has the quadratic form

$$V(z) = (z - z^\star)^\top P (z - z^\star),$$

where $P$ solves the discrete algebraic Riccati equation

$$P = Q + A^\top P A - A^\top P B \big(R + B^\top P B\big)^{-1} B^\top P A.$$

The target and input must be shifted to a feasible equilibrium; general affine tracking can add linear and constant terms. We will solve this DARE numerically for the exactly specified LQ model and test the resulting terminal value in a short-horizon planner.

Outside special cases such as LQR, a nonlinear problem's value function need not have this quadratic form and may lack a tractable closed form. It can be approximated, for example by a learned $V_\theta$. When it preserves decision-relevant value differences or gradients over candidate terminal states, it can support a shorter explicit rollout. Shortening that rollout reduces the maximum transition-Jacobian chain length and may mitigate horizon-induced vanishing or exploding sensitivities, although the terminal value can introduce its own conditioning problems. The model and value become coupled when joint training links them through shared parameters, representations, losses, gradients, or targets. Coupling also arises when value targets depend on model rollouts. An optimizer can push candidate terminal states outside the value function's training support. Bellman-style training may impose consistency constraints, but it does not remove approximation error. A learned terminal value therefore trades part of the omitted-tail error for value-approximation error while the shorter action problem still has to be optimized.

Differentiable value functions support at least three planning patterns.

As a terminal cost. Backpropagation starts from $\nabla V_\theta(z_H)$. That terminal-state adjoint propagates through the rollout and contributes a gradient to every earlier action.

For direct action optimization. If $Q_\theta(z,a)$ denotes a reward or return to maximize and is differentiable in a continuous action, one ascent iteration is

$$a^{(k+1)} = a^{(k)} + \eta \nabla_a Q_\theta\!\left(z,a^{(k)}\right),$$

with appropriate constraint handling for constrained actions, such as projection or reparameterization for simple action bounds. This $Q$ uses a reward convention, unlike the continuation cost $V$ above; for an action cost-to-go, the corresponding update uses the negative gradient. Each iteration differentiates one $Q$ evaluation rather than an explicit model rollout. Finding the global greedy action still requires solving a generally non-convex problem and may need restarts; differentiability alone guarantees neither a global maximizer nor even progress under an arbitrary step. For the displayed one-step update, only errors in the action gradient at the current iterate alter the step. Errors in action rankings can still affect candidate selection or the eventual greedy action.

As a tail estimate after finite lookahead. A terminal value can make short rollouts effective by estimating what lies beyond them. TD-MPC scores short latent rollouts with predicted rewards and a learned terminal value during sampling-based trajectory optimization, whereas MuZero backs up a value from a search-tree leaf. The planned chapters on MuZero and Value-Equivalent Models and TD-MPC and Control-Centric Representations will develop those distinct uses.

An imperfect learned component can create an exploitation surface when its error varies across planner-reachable candidates and changes objective values or derivatives. A vanilla optimizer sees the supplied values and derivatives, not an independent test of whether the surrogate remains valid. When a reachable model error creates a smooth direction toward lower predicted cost, gradient optimization can exploit it rapidly. It is not guaranteed to find every error, nor is it uniformly faster than sampling: local basins, flat regions, constraints, and initialization still matter.

Concretely, this shows up as:

• Plans that drive the latent state to a region where a learned cost is artificially low, decoding to something nonsensical.
• Terminal cost-to-go estimates that are too low in unexplored regions, making unsupported terminal states look artificially attractive.
• Reachable dynamics-model errors that make impossible trajectories look favorable. Directed gradients can refine such an exploit quickly once it lies in the current basin, although the comparison with sampling depends on the error landscape and budget. This is the model-exploitation failure introduced in Sampling-Based Planning and Model Predictive Control.

Several defenses against model exploitation have analogues for learned costs and values, but their uncertainty or support measures must be defined for the component they protect:

• Uncertainty-aware objectives. An ensemble penalty can discourage plans in regions of high disagreement. When disagreement is a validated proxy for decision-relevant epistemic uncertainty, high-disagreement regions can also carry elevated exploitation risk. Agreement can still hide shared systematic bias. PETS uses probabilistic ensembles and particles to propagate uncertainty during trajectory evaluation rather than adding this exact penalty; the planned chapter on probabilistic and ensemble model-based RL will compare these mechanisms.
• Smoothness constraints. Penalizing cost-gradient magnitude over a training distribution can discourage steep local variation there. It neither guarantees the removal of spurious minima nor, by itself, guarantees control of gradients at planner-reachable states outside that distribution.
• Conservative objectives. Some conservative offline-RL objectives, such as COMBO-style value regularizers, penalize Q-values under distributions chosen to emphasize out-of-support model-rollout tuples, aiming for conservative estimates there. Other conservative methods act through reward penalties or policy constraints.
• Trust regions in action space. A conventional trust region bounds each action-sequence update around the current optimizer iterate. To bound cumulative drift from an initially trusted plan, impose a constraint relative to that fixed reference or an equivalent episode-wide deviation budget.
• Evaluation on the real system. No amount of surrogate engineering substitutes for checking whether the plan works on the real system.

A useful mental model is that ordinary optimization against a fixed learned surrogate can behave like an adversarial search for errors inside the optimizer's basin, even though it is not a formal min-max problem. With other factors controlled, gradients that align better with a locally valid descent direction and have usable scale and variance can make local surrogate optimization more effective. That can improve task performance or expose smooth model errors more aggressively. Optimization quality and real-system performance must therefore be evaluated separately.