Predictive Embeddings, JEPA, and Energy-Based Models › Predicting Representations Instead of Pixels

Start with the most basic design question. A world model is a function that takes what the agent has seen and done and produces something that stands in for what comes next. The question is what that "something" should be. In the observation-prediction view, that "something" is an observation, predicted by a separate observation-space model $h_\psi$: $\hat{o}_{t+1} = h_\psi(o_{\le t}, a_{\le t})$, where $\psi$ are the parameters of the observation-space model and $\hat{o}_{t+1}$ is the predicted next observation. This is the classical prediction problem: you predict the quantity you measured, and you are graded on how closely you reproduced it. In the representation-prediction view, we interpose an encoder $f_\theta : \mathcal{O} \to \mathcal{Z}$ and predict in $\mathcal{Z}$ instead:

$$\hat{z}_{t+1} = g_\phi\big(f_\theta(o_{\le t}), a_{\le t}\big), \qquad \mathcal{L} = \big\| \hat{z}_{t+1} - f_{\bar\theta}(o_{t+1}) \big\|_2^2 .$$

where:

• $\hat{z}_{t+1}$: the predicted latent at time $t+1$
• $g_\phi$: the dynamics model (or predictor) that maps the context embedding and actions to a predicted latent
• $\phi$: the parameters of the dynamics model (or predictor)
• $f_\theta$: the online encoder that maps an observation sequence (or context) to a latent, parameterized by $\theta$
• $\theta$: the parameters of the online encoder
• $\mathcal{O}$: the observation space
• $\mathcal{Z}$: the latent embedding space
• $o_{\le t}$: the sequence of observations up to time $t$
• $a_{\le t}$: the sequence of actions up to time $t$
• $\mathcal{L}$: the loss function
• $f_{\bar\theta}$: the target encoder, a separate copy of the encoder with parameters $\bar\theta$
• $\bar\theta$: the parameters of the target encoder
• $o_{t+1}$: the observation at the next time step
• $\|\cdot\|_2^2$: squared Euclidean norm
• $t$: the current time step

The notation already contains the seeds of the whole chapter. There are two encoders, $f_\theta$ and $f_{\bar\theta}$, not one. The second one, the target encoder, is what makes the objective meaningful, and we will spend a full section on it. For now, notice what has changed structurally: the prediction target is no longer the measured observation itself. It is a learned embedding vector expressed in the encoder's coordinate system, and the meaning of "predicting well" depends entirely on what that coordinate system chose to preserve. Two different encoders trained on the same data can yield the same numerical loss while preserving completely different information, because the loss only measures agreement in whatever space the encoders happen to occupy.

The cleanest way to see why this matters is with an information budget. Let $S_t$ be a state sufficient to determine the conditional distribution of future observations and rewards given future actions, as discussed in Part II: Inference, Decisions, and Control, and let $O_t$ be the observation. Observations can contain large amounts of nuisance or redundant information even when the task-relevant state is compact. For a predictive world model, the representation should retain information needed to predict task-relevant future distributions under candidate actions. Modeling information with no downstream use can consume finite capacity and optimization effort without improving the decision objective.

An observation-reconstruction objective pressures $z_t = f_\theta(o_{\le t})$ to retain information about $O_t$. Perfect reconstruction by a deterministic decoder would imply that the observation can be recovered from the latent; finite-capacity, lossy training provides no such guarantee. In either case, minimizing observation error can spend capacity on pixels that do not help a decision. There is no general guarantee that a reconstruction-optimal representation is aligned with a decision-optimal one. The reconstruction objective does not know which pixels matter; it treats a trembling leaf and a braking car with the same weight, because both contribute to the pixel error in the same way.

A latent-prediction objective with a target encoder instead pushes toward a different quantity. If $z_t$ and $z_{t+1}$ are produced by encoders that must be mutually predictive, the encoder has an incentive to discard whatever is unpredictable between consecutive observations, because carrying unpredictable information only inflates the loss. If a pixel's next flicker value is independent of the entire available context, that value is unpredictable from preceding frames. Same-observation reconstruction can preserve and reproduce the realized flicker; future prediction may drop it. In this sense the prediction objective can perform automatic feature selection driven by temporal structure rather than explicit supervision.

This is not a free lunch. Discarding what does not help at the trained timescale can remove variables whose decision-relevant effects or dependencies appear only farther into the future. A pedestrian's crossing intention may become clear only after several observations; the consequences of wheel slip may also emerge over a longer rollout. Prediction horizon and masking directly shape which content the objective rewards. The target-encoder timescale is instead an optimization choice controlled by the EMA hyperparameter; it can influence the learned representation through training dynamics without directly selecting semantic content.

It helps to lay out the design space before diving into any one method. Latent-space prediction methods differ in what pressures keep the representation informative. Without one of these anchors or another effective constraint, a constant representation remains an available optimum, and training can collapse to it. The table separates explicit-negative contrast from the broader energy-based framework and separates architectural asymmetry from statistical regularization.

Reconstruction trains a decoder to recover observation content to the fidelity imposed by its loss; only perfect reconstruction with a deterministic decoder implies exact recoverability. This tethers the latent to the observation but can reintroduce pressure and cost from predicting nuisance detail. Explicit-negative contrastive objectives avoid the decoder by requiring compatible pairs to score above sampled decoys; their behavior depends on the loss and negative-sampling scheme. Asymmetric methods use combinations such as a predictor plus stop-gradient, often with an EMA target. Statistical methods instead penalize low variance or redundant coordinates. These mechanisms can discourage collapse, but none is an architecture-independent guarantee. Energy-based modeling is the broader scalar-compatibility framework: explicit-negative contrast is one way to train an energy, not the definition of the framework.

Handing the model freedom over the target representation has three concrete costs, and it is worth naming them now so the rest of the chapter reads as a response to them rather than a catalogue. These costs are not incidental; they are the reason that reconstruction and contrastive methods remain in active use despite the practical advantages of the asymmetric family.

The first cost is loss of an observation likelihood. If there is no normalized observation model, you cannot compute $p(o_{t+1} \mid o_{\le t})$. You can compute distances or energies between embeddings and train classifiers or planners on top of them, but those scores do not say how probable a particular observation was. This matters when normalized, calibrated probabilities or likelihood-based anomaly scores are required. Distance- and energy-based anomaly detection remains possible, but it needs separate validation or calibration.

The second cost is opacity of the prediction target. When a diffusion model predicts a frame, you can look at the frame. When a JEPA predicts an embedding, you interpret it through probes, nearest neighbors, or downstream task performance. Some semantic failures are therefore less immediately visible, and a non-degenerate embedding can still omit a variable the policy needs. The prediction loss does not know which variables the policy values unless the task construction makes them relevant.

The third cost is the trivial solution, which is the subject of the collapse section. A constant decoder cannot match varied observations, and a constant representation gives every contrastive candidate the same score. For an InfoNCE loss with $N$ candidates, equal scores give chance probability $1/N$ and loss $\log N$; that is not optimal whenever the model can score the positive above sampled negatives. Positive-pair matching can collapse because a constant output trivially matches a constant target, so its anti-collapse mechanisms must be evaluated explicitly.