Sufficient State and State Abstraction › Two Kinds of Sufficiency

The word "sufficient" hides a fork in the road. Before formalizing anything, let's separate the two standards that the rest of the chapter builds on. Both share the same skeleton: a quantity of interest computed from the full history should be recoverable from the compressed state. They differ in what that quantity is, and under limited data or capacity, optimizing one quantity can hurt the other.

A representation is predictively sufficient if it supports accurate forecasts of future observations. The intuition is straightforward: if you handed a forecaster the compressed state instead of the full history, its predictions of what comes next would be no worse. Everything the history knew about the future, the compression kept. Formally, let $z_t = \phi(h_t)$ be the encoded state. Predictive sufficiency means that for any horizon $T$ and any future action sequence $a_{t:t+T-1}$,

$$p\big(o_{t+1:t+T} \mid h_t, a_{t:t+T-1}\big) \;=\; p\big(o_{t+1:t+T} \mid z_t, a_{t:t+T-1}\big)$$

where:

• $h_t = (o_1, a_1, \dots, o_t)$: the full observation-action history available at time $t$
• $z_t = \phi(h_t)$: the encoded (compressed) state produced by the abstraction map $\phi$
• $a_{t:t+T-1}$: the sequence of actions $a_t, a_{t+1}, \dots, a_{t+T-1}$ (one action per transition step across the horizon) used for conditioning; predicting the $T$ future observations requires exactly these $T$ actions
• $o_{t+1:t+T}$: the block of future observations $o_{t+1}, \dots, o_{t+T}$ whose distribution we want to forecast
• $p(\cdot \mid \cdot)$: the conditional probability of the future observations under the environment's true dynamics

Two technical points about this definition deserve a pause. First, the condition holds for all horizons and all action sequences, including sequences the current policy does not visit. A representation that predicts well under the current behavior policy but fails under a different action pattern is not predictively sufficient; it has memorized a slice of the future rather than summarized the dynamics. Second, $a_{t:t+T-1}$ is treated as an externally fixed intervention. The compressed state must support forecasts under actions the agent has not yet taken, rather than merely reproduce correlations under the behavior policy.

The equality says the compressed state $z_t$ carries the same information about future observations as the entire history $h_t$: conditioning on the full history adds nothing once $z_t$ is known. All information about the future observations that was present in the history survives the compression. This is the Markov condition from Part II Chapter 1, restated as a property of a learned map: $z_t$ is a sufficient statistic for the observation-process future. Read it as a statistical statement about information flow: the map $\phi$ may collapse histories, but it collapses only along directions that the future distribution cannot distinguish.

This is the standard that likelihood-trained world models target, although finite data, finite capacity, and imperfect optimization do not guarantee that they attain it. Maximizing the likelihood of future observations pressures the latent state to encode anything that improves prediction, including billboard colors, background textures, and other features with zero bearing on the task. The objective asks whether a feature reduces prediction error, and the billboard does. The belief-state perspective of Part II Chapter 2 makes the same point constructively: for a known POMDP, the exact filtered belief $b_t$ is predictively sufficient, since it summarizes the observation history for forecasting under the model. It is a standard construction of a predictively sufficient state, though symmetries can permit further compression.

A representation is control-sufficient if it preserves everything needed to act optimally. The standard here is not "can you predict what you will see" but "can you decide what to do." Write $Q^*(h,a)$ for the optimal expected discounted return after taking action $a$ at history $h$ and acting optimally thereafter. A strong, directly checkable form of control sufficiency is

$$Q^*(h,a) \;=\; Q^*\big(\phi(h),a\big) \qquad \text{for every action } a,$$

where:

• $h$: a full observation-action history, the input to the optimal action-value function on the left-hand side
• $\phi(h)$: the compressed state obtained by applying the abstraction map to $h$
• $Q^*(\cdot,a)$: the expected discounted return after action $a$ and optimal continuation, applied to histories on the left and encoded states on the right

On the right-hand side $Q^*$ is applied in the abstracted space, so the notation ranges over histories on the left and encoded states on the right. Preserving every action value preserves their ordering and therefore the maximizing action set. A weaker policy-only definition asks only for an optimal policy that is a function of $z_t$ and attains the history-optimal return from every history; it need not preserve suboptimal action values. Preserving the scalar $V^*(h)$ alone would not be enough because two histories can have equal optimal values but require different actions. This chapter uses the stronger action-value definition unless stated otherwise.

This requirement is weaker than preserving the full future distribution because it constrains one action value per available action, not every observable outcome. Large amounts of information can be destroyed as long as all optimal action comparisons are preserved. It is still demanding because those action values solve a long-horizon optimization problem. The billboard fails the relevance test: it changes neither reward nor any reward-relevant transition, so no optimal action depends on it.

It is tempting to assume that a low prediction loss automatically yields good decisions, or that control-sufficient state is automatically a good predictor. The second implication fails even for exact sufficiency. The first requires more care: an exactly predictive state can support control when rewards are observable from the predicted trajectories, but a finite-capacity model trained on average prediction loss may still discard a small control-critical signal.

Control-sufficient without predictive sufficiency. Consider an agent moving down a corridor toward a reward at the far end. A state variable holding "steps remaining to the goal" supports the optimal policy perfectly. It predicts essentially nothing about the observations along the way: which posters hang on which walls, whether a door on the left is open. The agent can act optimally while forecasting little of the stream because the reward ignores most of it.

A predictive objective can miss control-critical information. A perfectly predictive state retains the cue in the following example; the failure comes from capacity and loss weighting, not from exact predictive sufficiency. Suppose one pixel in a $128 \times 128$ camera image turns red exactly one step before a reward becomes available. A model trained to minimize uniformly averaged pixel error gives that pixel one sixteen-thousandth of the aggregate weight, since a $128 \times 128$ image has $16{,}384$ pixels. Under limited capacity, the representation can discard it with little change in the aggregate score, crippling the downstream planner. A feature can be critical for control while contributing almost nothing to average prediction error. Flip the numbers around to see the asymmetry clearly: a feature occupying one pixel can be worth an entire reward, while a feature occupying half the screen, like the billboard, can be worth nothing. Average pixel error measures appearance, not consequences.

To make this asymmetry concrete before we formalize it, consider two features in a 128 x 128 camera stream: a billboard that fills half the frame and carries zero control consequence, and a one-pixel reward cue that gates the entire task. The next cell compares each feature's share of pixels, which is also its loss weight under equal per-pixel squared error, with its control relevance, measured as the normalized drop in optimal value if the feature is discarded.