Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

TD-MPC and Control-Centric Representations › Short-Horizon Planning with Terminal Values

Michael BrenndoerferJuly 17, 202610 min read

World Models HandbookAll chapters

Part 8 of 12Decision-Centric Research Lineages

Chapter 47TD-MPC and Control-Centric Representations › Short-Horizon Planning with Terminal Values

Reading section 4 of 9 · Short-Horizon Planning with Terminal Values · 10 minSection 4 of 9 · 10 min

Section 4 of 9 · 10 min read

TD-MPC and Control-Centric Representations › Short-Horizon Planning with Terminal ValuesOne section · Sections in this chapter4 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When can a short plan plus a terminal value outperform a longer model rollout? A good terminal value can replace many predicted steps, but its own errors then matter more. The choice is a measured tradeoff rather than a theorem that three steps beats thirty.

Over $H$ steps, an imperfect dynamics model feeds its predicted state into the next prediction. Errors may accumulate, cancel, or vary with the visited states; neither mean-squared prediction error nor return variance is guaranteed to increase at every horizon. Longer rollouts can also search more reward opportunities. Their net effect has to be evaluated on held-out trajectories and in closed-loop control.

If the critic's error at the reached terminal state-action pair is $\epsilon_Q(\hat z_H,a_H)$ , its direct contribution to the planning score is $\gamma^H\epsilon_Q(\hat z_H,a_H)$ . The discount attenuates a fixed error, but the terminal state itself changes with $H$ and may enter a region where critic error is larger. One critic evaluation is computationally cheap; that does not make its error constant in horizon.

The direct critic term shrinks with $\gamma^H$ only under the fixed-error comparison; model error may grow with $H$ . Those tendencies can create an interior optimum, but they can also produce a boundary optimum or a nonmonotone curve. If the critic is weak, planning longer helps only if the model remains accurate enough and the added search cost is worthwhile. Horizon selection belongs to validation, not a rule of thumb based on critic quality alone.

The next plot deliberately assumes one increasing model-error component and one fixed critic-error magnitude discounted by $\gamma^H$ . Under these chosen functions, their sum has an interior minimum. It is a hypothetical error budget, not a measured TD-MPC curve or a general upper bound on planning error.

In[7]:

Code

## Illustrative error decomposition for the horizon tradeoff (deterministic, no sampling).
## Assumed model-error term rises exponentially over this finite range.
## The terminal value's bias is fixed while its weight gamma**H decays.
h_grid = np.arange(1, 16)
gamma_tradeoff = 0.9
model_scale, model_rate = 0.2, 0.2
model_error = model_scale * (np.exp(model_rate * h_grid) - 1.0)
value_bias = 1.2
value_error = (gamma_tradeoff**h_grid) * value_bias
total_error = model_error + value_error
best_h = int(h_grid[int(np.argmin(total_error))])

Out[8]:

Visualization

Line chart of model error rising and value error falling, with a U-shaped total. — Hypothetical error budget with an assumed rising model term and a fixed critic-error magnitude weighted by gamma to the horizon. Their sum has an interior minimum for these chosen parameters; neither the shape nor the minimizing horizon is guaranteed for a trained agent.

There is also a classical control-theoretic reading. With exact dynamics and the exact optimal cost-to-go as terminal cost, optimizing the first action in a one-step plan recovers the optimal action by the Bellman equation. This requires the optimal terminal state value $V^*(s)=\max_a Q^*(s,a)$ , equivalently exact $Q^*$ evaluated at an optimal terminal action; exact dynamics and $Q^*$ with a suboptimal terminal policy do not suffice. Stability claims for approximate MPC require additional assumptions; an arbitrary terminal cost that upper-bounds cost-to-go is not by itself a sufficient blanket condition. The worked example below isolates the oracle-value case before the neural approximation complicates it.

Worked example: one-step planning with an exact terminal valueLink Copied

Make the argument concrete with a problem we can solve by hand. Take a one-dimensional linear system with quadratic cost,

s_{t+1} = A s_t + B a_t, \qquad A = 1.0,\; B = 0.5, \qquad r(s, a) = -(s^2 + a^2),

with discount factor $\gamma = 0.9$ . For this linear-quadratic problem the optimal value function is exactly quadratic, $V(s) = -P s^2$ , where $P$ solves the discounted Riccati equation

P = 1 + \gamma A^2 P - \frac{\gamma^2 A^2 B^2 P^2}{1 + \gamma B^2 P}.

The Riccati equation is the fixed point of the Bellman equation for this problem: substituting a quadratic value function into the Bellman optimality condition and matching coefficients yields exactly this algebraic equation for $P$ . Substituting the numbers gives $P = 1 + 0.9P - \frac{0.2025 P^2}{1 + 0.225P}$ . Multiplying both sides by $(1 + 0.225P)$ and collecting terms yields

0.225 P^2 - 0.125 P - 1 = 0 \quad\Longrightarrow\quad P = \frac{0.125 + \sqrt{0.125^2 + 0.9}}{0.45} \approx 2.404 .

We take the positive root because $P$ must be positive for the value function to be a cost (negative reward) that increases in magnitude with the state. The optimal controller is linear, $a^*(s) = -0.702\,s$ , and the optimal return from $s = 1$ is $-P = -2.404$ .

Now run a one-step planner from $s = 1$ with $H = 1$ . Here we replace TD-MPC's learned-policy, twin-critic terminal score with the oracle $V^*(s)=Q^*(s,a^*(s))$ ; this is the exact optimal continuation, not a claim that any terminal policy action would work:

J(a) = -(1 + a^2) + \gamma V(A \cdot 1 + B a) = -1 - a^2 - 0.9 \cdot 2.404\,(1 + 0.5a)^2.

Expanding the square and collecting terms gives $J(a) = -3.1636 - 2.1636\,a - 1.5409\,a^2$ . Setting the derivative to zero yields $a = -0.702$ , and substituting back gives $J = -2.404$ . A one-step plan with the exact terminal value reproduces the infinite-horizon optimum exactly, both the action and the value.

Remove the terminal value and the one-step planner maximizes $-1-a^2$ at $a=0$ : do nothing now. If it follows that single myopic action and then switches to the optimal controller, its total return is $-1+\gamma V(1)\approx-3.164$ , a cost magnitude about 31.6% above the optimum $2.404$ . If it repeatedly chooses the myopic action, the state stays at $s=1$ and the return is $-\sum_{t=0}^\infty 0.9^t=-10$ . These are different continuation assumptions; the 31.6% number belongs only to the first.

The same argument becomes a picture if we plot the planning objective over actions. The terminal value moves the maximum away from the tempting "do nothing" action and onto the optimum. All quantities are computed from the closed-form worked example, so the curves are analytic and deterministic.

In[9]:

Code

## Analytic curves for the one-dimensional LQR worked example.
## V(s) = -P s**2 with P the positive Riccati root; s_0 = 1, A = 1, B = 0.5, gamma = 0.9.
lqr_gamma = 0.9
P_pos = (0.125 + np.sqrt(0.125**2 + 0.9)) / 0.45
s0 = 1.0
a_grid = np.linspace(-1.5, 1.5, 401)


def _j_with_terminal(a):
    s_next = 1.0 * s0 + 0.5 * a
    return -(s0**2 + a**2) + lqr_gamma * (-P_pos * s_next**2)


def _j_myopic(a):
    return -(s0**2 + a**2)


j_terminal = _j_with_terminal(a_grid)
j_myopic = _j_myopic(a_grid)
a_star_terminal = float(
    -lqr_gamma * 0.5 * P_pos / (1.0 + lqr_gamma * 0.5**2 * P_pos)
)
a_star_myopic = float(a_grid[int(np.argmax(j_myopic))])

Out[10]:

Visualization

Two curves of planning objective versus action, peaking at different actions. — One-step objectives for the exact linear-quadratic example at s = 1. The analytic maxima are a = 0 without a terminal value and approximately a = -0.702 with the exact optimal terminal value. The curves are different objectives; their vertical separation is not a performance-gap estimate.

Three lessons fall out of this exercise, and all three survive into the neural setting:

With exact dynamics and the exact optimal terminal value (equivalently, exact $Q^*$ at an optimal terminal action), one-step maximization gives the optimal action. A suboptimal terminal action, approximate dynamics, or approximate values can break that equivalence.
Myopic planning is not merely suboptimal in magnitude; it systematically prefers actions that look good now because it cannot see the consequence. The wrongness is structured, not random.
The terminal value can change the maximizing action. For an approximate critic, its score error enters with a factor $\gamma^H$ and can alter action ranking; the size of that effect depends on the competing score gaps.

The exact example explains why terminal estimates can matter, not how accurately a learned critic will perform. A short horizon shifts responsibility to the critic; whether that helps depends on the quality of both components near the candidate trajectories.

Choosing the horizon in practiceLink Copied

The horizon is a hyperparameter with a real tradeoff. Use open-loop prediction checks to identify model weaknesses, then test candidate horizons in closed-loop control. A terminal-value ablation answers a different question.

Open-loop error by horizon. Roll the learned model forward with recorded actions from held-out trajectories and compare predictions with observed rewards and encoded successor states. Flat average error alone does not certify that a longer planner is safe: the planner may select different actions or exploit rare optimistic errors.

Controlled horizon sweep. With trained weights, terminal-value setting, initial states, seeds, action-repeat setting, and evaluation budget fixed, compare closed-loop returns across several horizons. The best horizon may differ from the one with lowest recorded-action prediction error because the planner selects its own actions. Include inference latency when the controller has a deadline.

Terminal-value ablation. At a fixed horizon, remove the terminal value while holding weights and evaluation conditions fixed. This isolates its contribution for the trained agent; it does not select a horizon. A small gap can mean the value estimate is unnecessary or that it is not yet useful.

Open-loop prediction quality, horizon-dependent closed-loop return, and terminal-value contribution are distinct measurements. The toy below prints the first and two small closed-loop comparisons, but its three evaluation starts are far too few for a reliable horizon choice.

One more practical consideration: horizon and action repeat interact. If the environment repeats each action for several simulator steps, a horizon of $H$ covers $H$ times as much real time, and $H = 3$ can correspond to a substantial fraction of a second. Report horizons in units of real time when comparing across papers. A horizon of three in a setup with action repeat of four is really a twelve-step horizon in simulator time, and comparing it to a paper with no action repeat would be misleading.