Sufficient State and State Abstraction › Limitations and Impact

The exact constructions in this chapter are directly computable for small, known MDPs. They require access to the reward and transition model, and dense metric procedures update at least a quadratic number of state pairs. Continuous or high-dimensional systems require additional structure, numerical approximation, or learned surrogates rather than tabular enumeration. Those approximations reintroduce error: value bounds depend on reward, transition, and metric estimation accuracy, and a learned representation can merge states whose relevant difference was absent from its training signal.

Task-awareness introduces its own fragility. Control-sufficiency is defined relative to a reward, and a representation distilled for one reward has no obligation to serve another. Worse, on-policy approximations inherit the current policy's blindness: if the policy never opens a particular door, the metric sees no behavioral difference behind it and may collapse those states, exactly erasing the information a future policy improvement would need. This is the representation-learning face of the model-exploitation problem from Part II Chapter 6: the agent's abstractions and its behavior co-evolve, and each can carve away information the other will later want. The dynamics are worth spelling out. A policy shapes the data distribution; the data distribution shapes the metric or value signal; the signal shapes the abstraction; and the abstraction constrains what the next policy iteration can even perceive. A loop with that structure can spiral: each cycle of policy improvement and representation refit removes a little more of the evidence that improvement depends on. Practical systems respond with optimistic exploration around abstraction boundaries, periodic re-fitting of the metric as the policy changes, or deliberately over-complete representations when the task family is uncertain, but each remedy trades away some of the compression that made abstraction attractive.

There is also an evaluation gap worth taking seriously. We can verify exact bisimulation on a known MDP, but for a learned representation in a complex environment we have no ground-truth partition to compare against. The practical criteria are indirect, and each has blind spots. You can check closed-loop performance, but a representation can score well on the training task while being fragile one task over, since training-task performance only audits the one consumer the training loop exercised. You can check invariance under augmentations, but that only audits the nuisances you thought to enumerate, and the dangerous nuisances are by definition the ones you did not. You can check value consistency across held-out states, but a value head can mask gaps in the underlying dynamics model, producing accurate-looking values built on a distorted internal geography. Multi-step rollout fidelity in the abstracted space, planning speed and quality in the quotient, and transfer to new reward functions form a more honest battery, and we will develop evaluation methodology systematically in Part XI.

These ideas give a precise answer to "what may a world model forget." When an exact quotient has far fewer states, each planning backup operates on the smaller quotient while preserving the action values guaranteed by bisimulation. Learned metrics and decision-focused latent models extend the same question to visual control, but without the exact certificate unless approximation error is controlled. The predictive-versus-control distinction also gives a practical diagnostic: when a world model predicts well and controls poorly, inspect whether its loss preserved the signals the planner consumes.