Predictive Embeddings, JEPA, and Energy-Based Models › Summary
Michael BrenndoerferJune 29, 20262 min read
World Models HandbookAll chapters Part 5 of 12World-Model Architectures
Predictive Embeddings, JEPA, and Energy-Based Models › SummaryOne section · Sections in this chapter9 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
- Predicting representations instead of pixels shifts the model's accountability from rendering the world to capturing what the target construction rewards. It can discard unrewarded nuisance detail and information that is unpredictable at the trained timescale, but neither outcome is guaranteed; auxiliary objectives can reward retaining either.
- The EMA-target JEPA recipe studied here combines a context encoder, an EMA target encoder, a predictor, and detached targets. Generic JEPA does not mandate this exact recipe, and there is no observation decoder.
- In a simplified linear fixed-point analysis, a learnable predictor can absorb part of the context-to-target regression and permit nonzero stationary solutions. It is an important optimization component, not a general mathematical guarantee against collapse.
- Energy functions provide a common compatibility language. Maximum-likelihood EBM training has positive and negative phases; practical methods either approximate the model expectation or replace maximum likelihood with objectives such as sampled classification, margin ranking, or score matching.
- Contrastive estimation uses explicit decoys. Asymmetric methods use stop-gradient and, in some recipes, a slowly updated target to alter positive-pair optimization dynamics; they do not estimate the maximum-likelihood negative phase.
- Complete collapse maps every input to a constant, while dimensional collapse concentrates variation in a low-effective-rank subspace. Effective rank and per-dimension variance diagnose degeneracy. A held-out linear probe tests whether one chosen variable is linearly recoverable; task usefulness requires downstream evaluation.
- Anti-collapse mechanisms fall into three families: contrastive negatives, architectural asymmetry (predictor, stop-gradient, EMA target), and variance or covariance regularization. They compose, and the worked example uses two at once.
- The costs include no direct observation likelihood, no task-agnostic sufficiency guarantee, and the weakness of point predictions for multimodal futures. The next chapter examines hierarchical, hybrid, omnimodal, and modular architectures, including multimodal systems that combine reconstruction and predictive objectives.