Latent-Variable Representations › Latent-Variable Representations

Imagine a robot with a camera pointed at a pendulum. The camera delivers 16 by 16 grayscale images, 256 numbers per frame, yet each snapshot lies on a one-dimensional manifold parameterized by the pendulum's angle. Modeling the raw frames exposes a dynamics model to a much higher-dimensional observation representation than that underlying snapshot manifold. It must learn or preserve the renderer, including the bob's glow, the fixed background, and the correlations among pixels, in addition to the pendulum's dynamics. What the robot needs is a compact internal description, a latent state, that retains the information required for prediction and control. When observation generation is part of the model, a decoder maps that state to a distribution over observations; the state need not preserve every detail of the particular input frame.

Chapter 2 of this part asked what makes a state representation sufficient for control. This chapter asks a harder, more practical question: when nobody hands you the state, how do you learn it from raw observations? Latent-variable models are one influential family of approaches in modern world modeling. You will learn how autoencoders compress observations, why plain reconstruction is a trap, how variational inference turns compression into probabilistic modeling, why bottlenecks can collapse, how to work with discrete codes, and what makes one latent geometry better than another.

The throughline is a single idea: a good latent representation keeps the information the world model needs and discards the rest on purpose. A reconstruction-only autoencoder selects information by reconstruction utility, not by downstream decision relevance, so useful and ornamental factors may remain mixed. The machinery we build in this chapter, from the ELBO's regularization term to the bottleneck coefficient $\beta$ to the discrete codes of a VQ-VAE, gives us ways to control and audit what the latent retains. None of these tools identifies task relevance automatically; that still has to be measured downstream.