Differentiable Planning › Worked Example: A Double Integrator

We use a problem whose action gradient and optimum have closed matrix forms that we can verify numerically. The continuous dynamics are $\dot{x}=v$ and $\dot{v}=a$, with acceleration held constant over each sampling interval. Their exact zero-order-hold discretization at time step $\Delta t$ is

$$z_t = \begin{bmatrix} x_t \\ v_t \end{bmatrix}, \qquad z_{t+1} = \underbrace{\begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix}}_{A} z_t + \underbrace{\begin{bmatrix} \tfrac{1}{2}\Delta t^2 \\ \Delta t \end{bmatrix}}_{B} a_t.$$

The cost is quadratic with target $g$:

$$J(a) = \sum_{t=0}^{H} (z_t - g)^\top Q (z_t - g) + \sum_{t=0}^{H-1} R a_t^2.$$

This linear-quadratic structure makes the action gradient affine and the unconstrained optimum available from a linear solve. It is therefore a useful test case: we can compare autodiff with an independently assembled quadratic gradient at a declared action vector.

Because the dynamics are linear, the state at time $k$ is an affine function of the whole action sequence:

$$z_k = A^k z_0 + \sum_{j=0}^{k-1} A^{k-1-j} B \, a_j = A^k z_0 + G_k a,$$

where $a \in \mathbb{R}^H$ collects the actions and $G_k \in \mathbb{R}^{2 \times H}$ has columns

$$[G_k]_{:,j} = \begin{cases} A^{k-1-j} B, & j \le k-1, \\ 0, & j \ge k. \end{cases}$$

Substituting into the cost and collecting terms gives a quadratic function of $a$:

$$J(a) = a^\top M a + 2 a^\top b + \text{const}, \qquad M = \sum_{k=0}^{H} G_k^\top Q_k G_k + R I_H, \qquad b = \sum_{k=0}^{H} G_k^\top Q_k \big(A^k z_0 - g\big),$$

with $Q_k=Q$ for all $k$ in this example. The gradient is affine in the actions:

$$\nabla_a J = 2 M a + 2 b.$$

Three consequences follow. The conditioning and nonlinear-gradient lessons extend beyond this toy problem, while strict convexity and the unique global optimum rely on this linear-quadratic structure with $R>0$.

The objective is strictly convex. $M$ is a sum of positive semidefinite terms plus $RI_H$, so $M\succ0$ when $R>0$. The unique local minimum is the global minimum,

$$a^\star=-M^{-1}b.$$

With exact gradients and a suitable step size, gradient descent converges to it; an excessive fixed step can still diverge.

For a suitably scaled fixed step, the convergence bound is governed by the condition number of $M$. Near the optimum, equal displacements along eigenvectors with small eigenvalues raise the cost less than displacements along steep eigenvectors. A single stable scalar step then contracts slowly along the flat directions. For this double integrator, the eigenvalue spread grows over the horizons we test; that trend is not universal.

For nonlinear multi-step dynamics, the gradient is generally not affine in the actions. Pendulum, cart-pole, and neural-dynamics planning objectives are often non-convex, although special nonlinear cases can remain convex. If the objective is bounded below and has an $L$-Lipschitz gradient, fixed-step gradient descent with $0<\alpha<2/L$ drives the gradient norm toward zero. Any convergent iterate sequence therefore converges to a stationary point, which need not be a local minimum. The result can depend strongly on initialization, and warm starts can help retain a useful basin when consecutive problems change smoothly.