Pretrained Visual Models and DINO-WM › Generalization with Pretrained Encoders

The central hypothesis behind DINO-WM is that a frozen, general-purpose visual encoder can support useful transfer across goals and scene variation. Whether it outperforms a task-specific representation depends on the task, data, and baseline. It helps to separate the forms of generalization rather than treating them as one result.

Visual generalization. Pretraining on diverse images can help an encoder recognize useful structure outside the dynamics dataset. It does not guarantee stability under a new camera, lighting condition, wallpaper, or room. Measure both feature changes and closed-loop performance under the shifts that matter for deployment.

Task generalization. A goal image can change the target without retraining the dynamics model, provided the new goal is observable in the representation and reachable through actions covered by training. In the cursor example, different target positions fit that pattern. Arbitrary goals do not.

Structural information. Patch tokens retain positions in a grid, so corresponding-token costs can reflect local differences. That is not the same as an explicit keypoint matcher: a moving object may shift across token indices, and appearance or occlusion can break naive correspondence.

What does not come for free is equally important.

New dynamics are not free. A visual encoder alone says nothing about how actions change the scene. A predictor trained on rigid-object motion may need additional data or adaptation for cloth; whether the encoder remains useful is an empirical question.

New observation regimes are not free. Camera, resolution, field-of-view, and modality shifts can change features and transition statistics. A natural-image RGB backbone should not be assumed to work on depth, thermal, or fisheye imagery without testing or adaptation. This is the distribution-shift problem discussed in Part VI: Learning World Models at Scale, now at the encoder.

Encoder bias can be a ceiling. If a frozen encoder maps two task-distinct observations to exactly the same current features and supplies no distinguishing feature history or other input, a feature-only predictor cannot recover the difference. Near-invariance is less absolute: a head might amplify retained signal, while a new sensor, history, or backbone adaptation may be needed if the distinction is truly lost. Prediction loss alone may not expose this failure.

The practical consequence is that frozen-feature world models should be evaluated on closed-loop behavior, and qualitatively on whether the encoder's features respond to the variables your task cares about. A quick diagnostic is to collect a batch of states that differ only in one task-relevant factor, encode them, and look at how much the features move. If that movement is barely detectable relative to nuisance variation or prediction noise, a low training loss offers little reassurance about closed-loop performance.

Finally, “frozen” admits intermediate designs. A trainable adapter atop a fixed backbone can reweight information that the backbone retained, but cannot reconstruct information discarded completely. Whether to adapt the backbone itself depends on data, compute, and the task-relevant sensitivity tests above; there is no universal episode-count threshold.