Predictive Embeddings, JEPA, and Energy-Based Models › Predictive Embeddings, JEPA, and Energy-Based Models

Imagine a world model for a delivery robot navigating a sidewalk. To decide whether to step left or right, the robot does not need to know the exact pixel values of an oak leaf trembling in the wind, the precise visual texture of wet concrete, or the sub-millimeter position of every pebble. It needs to know that the pedestrian ahead is drifting right at walking speed, that the gap between the parked car and the curb is narrowing, and that its own left wheel is about to clip the curb. Much of the raw image can be irrelevant to this decision, and some of it can be unpredictable from the robot's context. A model that spends finite capacity reproducing the leaf and the concrete may be learning detail its controller will never use.

Every observation-generating architecture we have met so far in Part V: World-Model Architectures pays some price for that irrelevance. Reconstruction-based recurrent state-space models optimize an observation-reconstruction term, so they can allocate parameters and training signal to detail the agent will never use; how much depends on the architecture, data, and objective weights. Autoregressive token models predict the next discrete symbol, which means prediction errors in task-irrelevant regions still contribute to the training signal. Diffusion models typically generate by iteratively reversing a noising process, while flow models integrate a learned vector field from a base distribution to the data distribution. Both observation-generating approaches spend computation on relevant and irrelevant visual detail.

This chapter takes a different stance. Instead of asking the model to reproduce the world, we ask it to predict a representation of the world. The model never renders an image. It maps observations into a learned embedding, which may be made compact by architectural or capacity constraints; in temporal or action-conditioned versions, a predictor learns dynamics in that embedding space. The prediction target is a vector in a learned latent space. No decoder, no pixels, no per-symbol likelihood. The model's job is to answer a narrower question: given its context, what target representation is consistent with it?

The idea immediately runs into a serious problem. If the model chooses both the representation and the prediction target, it can cheat. Map every observation to the constant zero vector and the prediction is perfect. The loss goes to zero, the model learns nothing, and the agent is blind. An unprotected joint-embedding matching objective admits this constant solution, and standard symmetric implementations can collapse rapidly after training begins. The main anti-collapse responses in this chapter come in three forms:

• Explicit-negative contrastive methods, which train compatible pairs against sampled decoys.
• Asymmetric methods, which change the two branches' gradient updates and can discourage collapse in specific training recipes without making the constant solution impossible.
• Regularized methods, which constrain embedding statistics; VICReg, for example, directly penalizes low per-coordinate standard deviation and off-diagonal covariance.

The Joint-Embedding Predictive Architecture, or JEPA, is a recent predictive-embedding framework introduced in a 2022 position paper. Practical variants combine predictors and asymmetric target handling; explicit variance or covariance regularization is a separate, optional mechanism. This chapter builds up the EMA-target structure found in I-JEPA and V-JEPA, studies a squared-error instantiation for its mathematical analysis and toy system, connects that illustrative discrepancy to the broader energy-based view, and compares collapsed and non-collapsed representations after training.

By the end you should be able to look at a "predict in latent space" objective and ask three questions: what discourages collapse, what kind of uncertainty the prediction target can express, and what evidence shows that the representation retains information an agent needs to act. Those questions expose most of the important design tradeoffs.