Pretrained Visual Models and DINO-WM › Frozen Visual Representations as State

The first design decision in a world model is what information the predictor uses as state. In Part III: Representing Agents and Worlds, predictive sufficiency meant that the future is conditionally independent of earlier history given that state and the actions. A frozen encoder supplies a per-frame embedding, not a guarantee of sufficiency. DINO-WM's transition model also consumes a short history of embeddings and actions; the one-frame model in our toy is an explicit simplification. The frozen visual features are not shaped by a task reward or value signal, but the learned predictor can aggregate their temporal context.

Formally, let $o_t$ be the observation (an image) and let $f_\theta$ be the encoder with parameters $\theta$ that are never updated. Define

$$z_t = f_\theta(o_t).$$

For a Vision Transformer with patch size $p$, we can retain a grid of patch tokens rather than pool them into one vector. The transformer divides a patch-divisible image into non-overlapping patches, embeds each patch, and lets attention mix information across positions. In the equation below, $H$ and $W$ are the dimensions after any padding and are divisible by $p$. Excluding any global or register tokens, the patch output has shape

$$z_t \in \mathbb{R}^{N \times D}, \qquad N = \frac{H}{p}\cdot\frac{W}{p}.$$

For DINOv2 ViT-S/14 at $224 \times 224$ resolution, that is $N = 256$ patch tokens of dimension $D = 384$: a $256 \times 384$ per-frame feature matrix. Prediction may still need earlier frames and actions.

Not every pretrained network produces a usable state space. The properties that matter for world modeling are subtly different from the properties that matter for classification. A classifier only needs to represent whatever distinguishes its label categories, whereas a world model needs to represent everything that changes in a way that affects the future. These are related but not identical demands, and a backbone that excels at one can be mediocre at the other.

• Informativeness. The encoding must preserve the distinctions that matter for predicting dynamics. If situations with different relevant futures produce the same complete predictor input, including the available feature history and actions, that predictor cannot separate them. A collision in one frame's features alone may be resolved by history or another sensor; freezing only blocks recovery of distinctions absent from every available encoded input. A fixed one-frame feature cost can still alias distinct goals.
• Smoothness. Small, task-relevant changes should ideally produce predictable feature changes. An erratic local encoding can make dynamics harder to learn, but smoothness alone does not make interpolation in feature space physically realizable.
• Spatial structure. Patch indices expose image location and can help a predictor exploit locality. Translation equivariance means that shifting an input shifts the output in the same way; a patch grid does not confer that property on a position-embedded Vision Transformer, and real motion need not be a translation.
• Stability across nuisance factors. Lighting, texture, exposure, and clutter may be irrelevant to one task but important to another; measure whether the feature response suppresses only the variation your task can safely ignore.
• Cost. A $256 \times 384$ patch set contains 98,304 scalar values, versus 150,528 RGB channel values at $224 \times 224$. That is only about a 1.53-fold reduction in raw element count, not proof of a storage or compute advantage. The possible benefit is in the representation's usefulness to prediction and planning, which must be measured rather than inferred from dimension alone.

Self-supervised objectives emphasize these properties differently; no universal ordering or necessary trade-off follows from the objectives alone. The choice of pretraining method is therefore not a detail you can ignore; it helps shape the state space your predictor will inhabit.

SimCLR and MoCo contrast augmented views of the same image against other examples. Their objectives encourage agreement under chosen crops and color changes; they do not guarantee exact invariance in the features later used for control. Strong color augmentation could make a red/blue distinction harder to use, for example, if color is the variable a robot needs. The augmentation family is therefore an assumption to probe against the downstream task.

MAE reconstructs masked image patches from visible-patch encodings. Its reconstruction objective allocates training effort to pixels, but that does not establish that its learned features are merely low-level or unsuitable for dynamics. A fair encoder comparison has to hold the predictor and control task fixed.

DINO trains a student to match an exponential-moving-average teacher across augmented crops, with centering and sharpening in its collapse-prevention recipe and no explicit negative pairs. DINOv2 builds on that lineage with an iBOT-style masked-patch objective and KoLeo regularization of class-token feature spread, among other changes. Its paper reports useful spatial and object-part cues, including qualitative correspondence examples; its semantic-segmentation benchmarks train readout heads, so they are not evidence of segmentation by raw nearest neighbors alone. DINO-WM uses DINOv2 patch features, but that does not make DINOv2 the default backbone for every feature-space world model.

Here is the tension stated plainly. A frozen encoder was optimized to make some distinctions and ignore others. Those choices were made by whoever pretrained it, for whatever data and objective they had in mind, with no knowledge of your task. The encoder is, in effect, a frozen hypothesis about which differences in the world are worth representing, and you inherit that hypothesis wholesale.

For a world model, it helps if nuisance changes have a small feature effect while controllable changes remain distinguishable and predictable. Invariance means the features are unchanged under a transformation; equivariance means they transform in a corresponding, structured way. Neither property is automatic from pretraining, and a search-based planner does not require every feature coordinate to change monotonically with an action. It needs a goal cost and dynamics model that preserve the task-relevant differences well enough to rank candidate actions.

In Part VIII, TD-MPC illustrated a representation shaped by task reward and value objectives. Frozen visual features take a different route: keep the representation fixed and fit the predictor in that space. Task-specific interaction data can favor adaptation; broad visual pretraining can make freezing attractive when such data are limited. Neither choice wins categorically. Encoder coverage, action data, dynamics, and the evaluation metric decide.

A useful first question is whether the task variable is visible in the available frames. Object location and scene layout may be; material compliance or hidden velocity may not be recoverable from one still image. Human ability to label a variable is not proof that DINOv2 retains it. Test pairs of states that differ in the variable, compare their features against nuisance variation, and use histories or other sensors when one frame is insufficient.

Often not. A Markov state makes the future independent of earlier history once the present state and the relevant actions are known. A single frame may leave velocity or an occluded object unresolved. A pendulum passing the same non-extreme angle with positive versus negative angular velocity looks the same in a still image but moves in opposite directions next. The one-frame embedding cannot resolve that ambiguity by itself.

There are three standard responses, all of which appear in practice:

• Frame stacking. Concatenate features from the last $k$ frames. This is a relatively simple way to expose short-term history; a short stack can sometimes disambiguate velocity and direction of motion.
• A recurrent or causal-attention predictor. Let the predictor consume a history of feature sets, so its internal state carries the missing information, as in Part V: World-Model Architectures. It can model interactions across a longer history, at the cost of more training and debugging complexity. For a fixed finite history, this is not categorically more expressive than a sufficiently capable model over stacked features.
• Accept the approximation. In a nearly memoryless, well-observed task, one frame may be sufficient in practice. The last action alone does not generally reveal velocity or an occluded object; measure the resulting prediction and control error rather than assuming a planner will tolerate it.

DINO-WM conditions its transition model on a short history of encoded frames and actions. Such context can reveal direction of motion without an explicit recurrent hidden state. Whether the chosen history is sufficient, and what it costs relative to another architecture, depends on the environment and should be measured.