Pretrained Visual Models and DINO-WM › Pretrained Visual Models and DINO-WM

A camera-frame predictor must account for appearance as well as motion. An unweighted pixel loss scores the wall, anti-aliased edges, and sensor noise alongside the object the agent must move. Depending on the architecture, modeling those details can use training or rollout compute without improving the decision. Higher resolution increases the number of predicted values; the fraction occupied by a task-relevant object shrinks only if the field of view or background grows relative to it. A photorealistic frame is therefore not, by itself, evidence of useful control dynamics.

DINOv2 offers a useful starting point: it was trained without labels on the 142-million-image LVD-142M corpus, and its patch features retain spatial and object-part cues in the authors' evaluations. Those cues are not a solved control state. DINO-WM tests a narrower proposition: freeze a pretrained visual encoder, then learn how its features change under recorded actions. This separates reusable visual pretraining from dynamics that still have to be learned for an environment and its action interface.

So DINO-WM splits the problem in two:

• A frozen pretrained visual encoder turns each observation into a set of feature vectors. Its weights do not change during dynamics training.
• A trainable predictor uses a history of those features and recorded actions to predict future features. The worked toy later simplifies this to one frame and one action.

The DINO-WM dynamics loss and planner need no image decoder or reward model; the paper trains an optional separate decoder to inspect predictions. Training still requires offline observation–action trajectories, which may include expert behavior, but a new feasible image goal needs no expert demonstration or reward label for that particular goal. Planning searches for actions whose predicted patch features approach the goal image's features. That distance is an explicit, imperfect task cost, not a reward function that appears automatically from pretraining.

This chapter explains when that separation helps, when it fails, and how to build a small version of it. It sits in Part IX: Foundation and World-Action Models after tokenized game and environment models, and prepares the ground for video generators and vision-language-action systems. Those later systems make different architectural choices; they do not all freeze a visual encoder or learn a separate small dynamics model.

The core trade is reuse against task adaptation. A frozen encoder gives the dynamics learner a fixed visual coordinate system and avoids learning that representation again from the environment's action data. It also limits what the predictor can infer from a frame: a distinction absent from the features cannot be recovered from that frame alone, although history or another sensor may help. The encoder is a hypothesis about which visual distinctions matter, and the rest of the pipeline depends on testing it.

We'll build the whole pipeline end to end on a tiny synthetic world so that every piece is inspectable. Along the way we'll examine three distinct failure modes: feature aliasing (task-distinct states look the same to the encoder), dynamics error (the learned predictor misjudges the effect of an action), and proxy misalignment (a low feature-space goal cost does not imply task success). The toy exposes these mechanisms, but it does not establish how often they occur with DINOv2 or on a robot.