TD-MPC and Control-Centric Representations › Limitations and Impact

One limitation is optimization against learned scores. The planner searches for sequences with high predicted reward and terminal $Q_\theta$; it may therefore favor errors in either the dynamics/reward model or critic, especially outside replay coverage. A short horizon restricts the number of model steps, but gives the terminal estimate more weight through $\gamma^H$ than a longer horizon would at fixed critic error. Policy-guided proposals and action sampling can affect exploration, not guarantee that elites remain in-distribution or that optimism disappears. Offline and Conservative Model-Based RL develops the related coverage problem.

Another limitation is architectural. Deterministic latent dynamics do not explicitly represent a distribution over next states. The value target uses a policy proposal instead of solving a continuous-action maximization, so proposal quality matters. The learned reward head is fitted to the task's scalar reward; changing the task objective generally requires adapting the learned components, not simply changing a prompt. These constraints distinguish this controller from the foundation-model lineage in Part IX.

For a risk-neutral objective with a well-defined, finite expected return, optimizing that expectation remains appropriate even when outcome tails are heavy. A deterministic model can still mis-rank actions if its single predicted path misses decision-relevant stochastic effects, and it cannot directly evaluate risk-sensitive criteria such as tail loss. A critic trained on real outcomes may capture some expected effects; explicit stochastic or ensemble models offer different tools when uncertainty itself matters.

Planning runs at each control step, so inference cost grows with candidate count, iterations, and horizon. A deployment with a tight control deadline must measure that cost. Reproducing either paper also requires its version-specific details: policy-proposal noise, target computation, update schedule, normalization layers, and, in TD-MPC2, distributional heads. The original planner does not z-score returns; our earlier illustration explicitly labels that operation as hypothetical.

The contribution is a practical division of labor: a decoder-free latent model supports short local search while a learned value estimates the tail. The original paper reports strong continuous-control results; TD-MPC2 later reports both 104 online single-task evaluations and a separately trained 80-task generalist with scaling across tested model sizes. These results support the approach on those benchmarks, while the earlier toy illustrates how easily a small implementation can fail to demonstrate the same outcome.