Predictive Embeddings, JEPA, and Energy-Based Models › Limitations and Impact

Reconstruction and explicit-negative contrastive learning are two prominent self-supervised approaches, alongside clustering, masked prediction, and other pretext tasks. Reconstruction provides an inspectable observation-space target and resists constant collapse through its decoder, but it can spend training signal on irrelevant detail. Explicit-negative objectives provide direct discrimination and mutual-information analyses, but depend on candidate construction and can motivate large batches or memory banks. Negative-free asymmetric methods avoid those candidate stores and observation decoders, while target encoders and predictors add their own computation. I-JEPA and V-JEPA demonstrate EMA-target predictive-embedding systems at image and video scale, with the nonsquared loss choices noted earlier. V-JEPA 2 further demonstrates action-conditioned latent post-training and robot planning after large-scale action-free video pretraining. Those results establish concrete uses; they do not isolate the objective from architecture, data, scale, masking, loss choice, or planning design as the cause.

The limitations cluster around likelihood, sufficiency, and multimodality. A basic predictive embedding can rank candidate futures by energy but cannot report a normalized probability for a specific observation. This removes direct likelihood-based anomaly scoring and probability calibration; energy or latent-error anomaly scores remain possible but require separate validation or calibration. Evaluation therefore uses explicit protocols such as held-out probes, task performance, retrieval, or rollouts, and conclusions can vary with the chosen protocol.

The basic JEPA objective also offers no task-agnostic sufficiency guarantee. An encoder can discard a variable that is hard to predict at the trained horizon even when its later effect matters for braking. Careful horizons, masks, conditioning, or task-aware constraints can reduce this risk, but guarantees require additional assumptions. Finally, squared error is a point objective: its conditional-mean prediction can be misleading for ambiguous futures. General high-dimensional energies can represent multiple modes, but sampling or minimization is often iterative unless the model has tractable structure or an amortized proposal. Some systems, including V-JEPA 2-AC, keep latent prediction deterministic and use it inside an action-sequence planner; that ranks actions but does not itself represent exogenous multimodal uncertainty.

The cost balance depends on architecture and execution phase. In common recipes, anti-collapse mechanisms can cost less than a large observation decoder or iterative sampler, but a target network still adds computation and state, and statistical losses add tunable weights. EMA timescale, variance regularization, predictor capacity, and prediction horizon can each affect training and should be validated jointly for the intended setup. An EMA decay that is too small for one recipe may reduce stability; an overly strong variance penalty can inflate coordinate variation without making it useful; a short horizon can miss variables whose decision-relevant effects appear later. Cheap per-coordinate statistics can run often, while effective-rank checks and trained probes can run at a cadence their cost permits. The worked example reports all three, but task-level evaluation remains necessary.

This family foregrounded a choice earlier world models also made: what the model predicts has consequences for the representation it learns. Reconstruction, explicit-negative discrimination, and asymmetric prediction hold a model accountable for different targets. The next chapter examines hierarchical, hybrid, omnimodal, and modular architectures, including multimodal systems that combine several training objectives. The same target-choice question carries into Part VI: Learning World Models at Scale.