TD-MPC and Control-Centric Representations › TD-MPC and Control-Centric Representations

Picture a quadruped learning to walk in a simulator. The controller must choose an action now, but the consequence that matters may arrive well beyond the next few steps. A learned model can help compare immediate consequences; rolling that imperfect model far into the future can make the comparison unreliable. TD-MPC makes the boundary explicit: plan briefly with the model, then use a learned value estimate for the rest.

One alternative learns a policy or action-value function directly from interaction without an explicit transition model. Another learns dynamics and searches through predicted trajectories. Their relative data and compute costs depend on the task and algorithm; the important distinction here is where prediction ends and value estimation begins.

Longer open-loop rollouts may expose a planner to model error and out-of-distribution states, but error need not rise monotonically on every trajectory or metric. The useful planning horizon therefore has to be measured, not deduced from a general slogan about compounding error.

TD-MPC, short for Temporal Difference Learning for Model Predictive Control (Hansen, Su, and Wang, ICML 2022), combines short-horizon latent-space planning with a terminal action-value estimate. Its reported continuous-control benchmarks show strong sample efficiency against the baselines tested. The model and critic are learned together: training unrolls predicted latent states for reward, consistency, and value losses, while the released implementation computes TD bootstrap targets from encoded observed successor states.

Two commitments define the method, and both are worth stating plainly before any notation appears.

1. The model should be control-centric. TD-MPC learns a decoder-free latent representation for reward prediction, short-horizon dynamics, and value estimation. No observation-reconstruction loss requires the latent to preserve every visual detail. This resembles the decision-relevant representation discussed in MuZero and Value-Equivalent Models, now with continuous actions and sampling-based rather than tree search.
2. The model should be trusted for a measured horizon. The reported TD-MPC experiments use a maximum planning horizon of five latent steps across tasks, scheduled upward from one early in training. A bootstrapped action-value estimate supplies the tail, analogous to the terminal value in an $n$-step return.

These choices shape both the method and its limitations. A latent without a decoder need not preserve enough detail to render an observation. A short plan relies more heavily on the critic, while still depending on the model's predictions along its short rollout. Both dependencies reappear in the training objective and failure modes below.

If you have read The Dreamer Family, you already know a world model that trains its latent by reconstruction and then trains an actor inside imagined rollouts. Sampling-Based Planning and Model Predictive Control introduced the MPPI and cross-entropy shooting lineage. TD-MPC combines sampling-based planning, value bootstrapping, and a decoder-free representation. MPPI searches over actions without differentiating through a learned rollout; temporal-difference learning estimates returns beyond the short plan; the representation is trained for control-related predictions rather than pixel fidelity.

First we build the model and separate its weighted multistep loss from policy optimization. Then we assemble the latent-space planner and work through an exact terminal-value example. Finally we examine TD-MPC2's 80-task generalist, run a small executable diagnostic that may fail to learn under its tiny budget, and trace the remaining failure modes.

Throughout, keep three things separate in your head: what the model predicts, how the planner uses those predictions, and how the value function summarizes what lies beyond. Most confusion about this family of methods comes from blurring those three roles. A model that predicts poorly may still support good decisions if the planner only relies on it briefly and the value function covers the rest. A value function that looks accurate in isolation may still be exploited by a planner that searches the edges of its support. Keeping prediction, planning, and value estimation conceptually distinct is what lets you diagnose which one is failing when the agent underperforms.