Latent-Variable Representations › Latent Geometry and Semantic Organization

Two latent spaces can match reconstruction error while differing enormously in usefulness for an agent. This section defines the geometric properties that matter, works through the linear case where the answer is exact, and states the identifiability limits of unsupervised representation learning.

Suppose two VAEs achieve identical reconstruction error. Their latent spaces can still differ enormously in usefulness for an agent. What distinguishes a good latent geometry?

• Supported coverage: avoid unintended gaps along the encoded data manifold, and interpolate along valid manifold paths. Topological holes, such as the interior of a circle, should be preserved rather than filled.
• Topological fidelity: the latent geometry should mirror the topology of the true state space. A pendulum's angle lives on a circle; a latent that flattens it onto a line creates a coordinate seam between the "ends" of the swing. An agent can plan discontinuously around a full rotation if its planner or dynamics model uses that nonperiodic metric without explicit wrap-around features, kernels, or boundary handling.
• Metric alignment: distances in latent space should be predictive of distances in world state, at least for the aspects the agent must control. Local differentiability or sufficient smoothness of the encoded transition or observation map enables local linearization. Metric alignment instead makes the Euclidean neighborhoods and step sizes used by that approximation meaningful for retrieval, planning, and control.
• Factor separation: when the downstream task requires independent intervention, editing, or interpretation, distinct world factors should be separable or independently traversable in latent space. That structure lets a planner vary one aspect of an imagined future without accidentally changing the others. Axis-aligned separation is not a universal requirement; an entangled representation can still be control-sufficient.

None of these properties is enforced by reconstruction alone, which is precisely why they deserve their own attention. The next two subsections make the claim concrete, first with an example where everything works out analytically, then with the theorem that bounds how far the good news extends.

To see that these claims are not hand-waving, consider the simplest possible latent-variable model: a Gaussian prior $z \sim \mathcal{N}(0, I_d)$ and a linear-Gaussian likelihood $p_\theta(x \mid z) = \mathcal{N}(W z + \mu, \sigma^2 I)$, where $x \in \mathbb{R}^n$. For this model the true posterior is analytically tractable, because the product of two Gaussians remains Gaussian and the integral in the marginal can be carried out exactly. Completing the square in the exponent of $p(z) \, p(x \mid z)$ gives

$$p(z \mid x) = \mathcal{N}(m, V), \qquad V = \big(I_d + \sigma^{-2} W^\top W\big)^{-1}, \qquad m = V\, \sigma^{-2} W^\top (x - \mu).$$

So the exact posterior mean is a fixed linear map and its covariance is constant. For centered data with $d<n$, let $\lambda_1 \ge \cdots \ge \lambda_n$ be the sample-covariance eigenvalues and suppose the retained eigenvalues satisfy $\lambda_d > \sigma^2$, where $\sigma^2$ is the fitted isotropic noise floor. At a global probabilistic-PCA optimum,

$$W = U_d\big(\Lambda_d - \sigma^2 I\big)^{1/2}R,$$

where $U_d$ contains leading sample-covariance eigenvectors, $\Lambda_d$ their eigenvalues, and $R$ is orthogonal. Thus the column space of $W$ spans a leading $d$-dimensional principal subspace. If an eigenvalue tie crosses the cutoff at $d$, the choice within that tied eigenspace is not unique; the theorem identifies the corresponding leading invariant subspace only up to that ambiguity.

A one-component PCA example illustrates the target subspace in that theorem. The next cell draws 300 points from an anisotropic Gaussian (seeded with np.random.default_rng(3)), computes the principal direction from the sample covariance, and records it together with the data for plotting. The scatter and the recovered axis are drawn in the figure cell below; the code computes PCA directly rather than fitting a VAE.

This PPCA result is due to Tipping and Bishop (1999); Lucas et al. (2019) analyze its connection to linear VAEs. It teaches two lessons.

First, the deep VAE extends the classical methods from the system identification chapter by replacing linear maps with nonlinear encoders and decoders. Gaussian factor analysis and PCA provide a rigorous anchor for the nonlinear case. When a nonlinear VAE behaves strangely, asking what the linear analogue would do can be a useful diagnostic.

Second, distinguish PCA eigenvectors from the latent loading basis. With distinct eigenvalues, PCA eigenvectors are fixed up to sign. In an isotropic-prior linear latent model, however, the loading matrix can be right-multiplied by an orthogonal rotation without changing the generative marginal likelihood. The principal subspace is identified, while equivalent latent coordinates inside it are not. Nonlinear latent models admit broader prior-preserving reparameterizations, so semantic coordinates still require additional constraints.

This arbitrariness has a formal version. Locatello et al. (2019) showed that unsupervised disentanglement is not identifiable without inductive biases. A prior-measure-preserving invertible transformation, or a consistent transformation of both prior and decoder with the appropriate change of variables, can leave the observation distribution unchanged while scrambling latent semantics. A latent space can look organized without factoring the world as hoped, and reconstruction quality alone cannot resolve the ambiguity.

The practical consequences for world model builders:

• Do not trust organization you did not measure. Probe latents against known generating factors when you have them. In simulation you can regress latent coordinates on true state variables and report how much variance each latent direction explains of each factor; if you cannot run this check, treat claims of disentanglement with suspicion.
• Exploit what you have. Actions, temporal consistency, and multi-view observations are inductive biases. Encouraging latents to be predictable from actions, or to change slowly, can reduce the family of equivalent representations left by pure reconstruction. Which transformations remain depends on the objective, data and action coverage, and model class. Each bias is an additional constraint that a reparametrization must preserve, and the more such constraints you impose, the smaller the family of equivalent solutions can become. This is precisely where the next chapter, on temporal state and memory, picks up.
• Judge latents by downstream control, not by aesthetics. A representation that is entangled but control-sufficient can be perfectly usable; a disentangled-looking one that hides the reward-relevant factor is not. The state abstraction criterion from the previous chapter, not the visual appeal of latent traversals, is the standard that matters.