Sufficient State and State Abstraction › Minimal State and Nuisance Invariance

Sufficiency alone does not make a representation good. The identity map is perfectly sufficient and perfectly useless. It keeps everything, predicts everything, decides everything, and compresses nothing. The second half of the design problem is minimality: among all sufficient representations, prefer the one that discards the most. Sufficiency sets a lower bound on what to keep; minimality sets a pressure to keep nothing beyond it. A good abstraction lives exactly at the intersection.

For exact bisimulation, minimality has a crisp answer within the class of bisimulation-respecting partitions. Collapsing each bisimulation class to a single point yields the coarsest exact quotient: any coarser partition violates reward agreement or successor-class agreement. This is not necessarily the smallest representation for one fixed policy or task, because non-bisimilar states can happen to share optimal actions. The number of quotient classes is an abstract-state cardinality, not a geometric latent dimension. In the corridor example, exact bisimulation yields five classes rather than ten because it removes the lamp bit while keeping the five behaviorally distinct positions.

This gives us a precise language for a failure mode we flagged in the previous section. Reconstruction loss rewards an encoder for retaining every feature the decoder can use, including billboards, background texture, and lighting. By itself, it does not target a minimal control-sufficient representation. With enough capacity, any lossless invertible encoding can minimize reconstruction loss; under a bottleneck, compression follows the weighting of reconstruction errors and regularization rather than control consequences.

The complement of sufficiency is invariance. Call a transformation of the observation a nuisance if it changes $o_t$ but not the sufficient state. The billboard color cycle is a nuisance. So are camera pose jitter, lighting changes, texture swaps on irrelevant objects, and the identity of visually distinct but dynamically identical objects. The unifying property is that the transformation moves pixels without moving consequences: after applying it, every reward, every transition probability, and every optimal decision is unchanged. Formally, if a group $G$ acts on observations and each $g \in G$ is a nuisance, we want our encoder, viewed as acting on single observations here (an extension of $\phi$ to histories), to satisfy

$$\phi(g \cdot o) \;=\; \phi(o) \quad \text{for all } g \in G,$$

where:

• $G$: a group of transformations acting on observations, each element of which is a nuisance (for example, rotations, color shifts, or lamp-color flips)
• $g \cdot o$: the observation $o$ after transformation $g$ has been applied
• $\phi$: the encoder mapping observations to the compressed state (here applied to single observations; in the predictive-sufficiency section it was applied to histories)

The group structure means transformations compose and invert. The equation makes every orbit $\{g \cdot o : g \in G\}$ lie inside one fiber of $\phi$, where a fiber is the set of observations mapped to the same encoded state. Group orbits need not exhaust behavioral equivalence: two observations can be interchangeable without being connected by a chosen transformation. For an ideal task-specific abstraction, each fiber should be a behavioral-equivalence class, while the nuisance orbits supply verified subsets that must remain inside those fibers.

Invariance is not free and not always wise. A transformation is a nuisance only relative to a task distribution. Cropping can be appropriate when camera translation truly leaves every task-relevant relation intact, but it is dangerous when reward depends on location, such as placing an object in a designated zone. Every augmentation is an implicit claim about what the controller may ignore. If that claim is wrong, no downstream function of the fixed invariant representation can recover the discarded evidence. Retraining the encoder from retained raw observations may restore it, but the current representation cannot.

The evidence lower bound used for a variational autoencoder contains a reconstruction term and a prior-matching term. Reconstruction error pressures the latent to carry details the decoder cannot predict from its other context. A conditional or autoregressive decoder can model some uncertainty without encoding every realized detail, so the pressure depends on decoder structure and likelihood choice. Increasing reconstruction weight still favors observation detail according to that likelihood, not control consequence, which creates the tension with task-specific abstraction.

This does not make such systems useless, and we will meet successful ones in the architecture chapters of Part V. It means their representations are abstractions only by side effect. Compression happens because the latent is small relative to the observation, so the model is forced to prioritize, but the prioritization follows reconstruction error, not control consequence. The remedy is to add terms that pull toward control relevance: reward prediction heads, value function auxiliary losses, or explicit bisimulation-shaped distances. Each of these adds a gradient pathway whose pressure is "encode this because it matters for decisions" rather than "encode this because it appears in the pixels." The system in Part III Chapter 3, which treats the latent state as a random variable with a learned prior, will give us the machinery for making this trade-off explicit in the objective.