Pretrained Visual Models and DINO-WM › A Worked Example: Reaching a Key from a Picture

Before writing code, let's walk through the whole pipeline on an example small enough to check by hand. This is the same structure the code implements, just with numbers chosen to make arithmetic possible. Working through the arithmetic by hand is worth the effort because it exposes exactly which assumptions the pipeline relies on and exactly where they break.

The world. A point cursor moves in the unit square. The state is the cursor position $s_t \in [0,1]^2$. The action $a_t \in \mathbb{R}^2$ is a displacement, and the dynamics are $s_{t+1} = \operatorname{clip}(s_t + a_t, 0.05, 0.95)$ with $\|a_t\|_\infty \le \delta$, where $\delta$ is the per-coordinate action limit. Each frame shows a red disk at a fixed key position $k$ and a blue disk at the cursor. The task is to put the cursor on the key. The analytic calculation below assumes the action does not hit the clipping boundary.

The encoder. Suppose, for the sake of the example, that the encoder produces a two-number feature vector, one number for each cursor coordinate:

$$z_t = \begin{bmatrix} \alpha\, s_t^{(x)} \\ \beta\, s_t^{(y)} \end{bmatrix}, \qquad \alpha, \beta > 0.$$

Real encoders are vastly more complex. This analytic feature map is invertible, so reaching zero feature distance is equivalent to reaching the physical goal. Invertibility alone does not preserve rankings of nonzero, weighted costs or guarantee that a finite-horizon planner finds the goal.

The dynamics. While clipping is inactive, the features evolve as

$$z_{t+1} = z_t + \begin{bmatrix} \alpha & 0 \\ 0 & \beta \end{bmatrix} a_t.$$

A residual predictor $F_\phi(z, a) = z + W a$ with $W = \mathrm{diag}(\alpha, \beta)$ recovers the interior transition exactly. A fitted $W$ can approach that matrix if the training set includes enough unclipped interior transitions to identify both action directions and noise is controlled; boundary-clipped samples do not obey this linear relation. One global linear map cannot represent clipping everywhere. This idealized encoder already supplies the two cursor coordinates, so this calculation does not establish that a real visual encoder will do the same.

The goal. The goal image is the picture with the cursor drawn on the key, so

$$z_g = \begin{bmatrix} \alpha\, k^{(x)} \\ \beta\, k^{(y)} \end{bmatrix}.$$

The plan. The toy example uses the summed cost $\sum_{k=1}^{H} \|\hat{z}_{t+k} - z_g\|^2$, unlike DINO-WM's terminal-state cost. If a feasible action sequence reaches zero feature residual, the analytic cursor is on the key. Neither invertibility nor this objective guarantees that a finite-horizon search finds that sequence, and nonzero plan rankings can differ from those under a physical-coordinate cost.

Limits of the analytic example. The rendered toy is rasterized and can map nearby physical states to identical frames before encoding; it can also occlude the red key beneath the cursor. Its learned encoder may discard more information, and its nonlinear dynamics model is only approximate. Finally, feature directions can receive unequal cost weights: if $\alpha$ is small, the analytic cost penalizes horizontal error weakly. These are distinct failure mechanisms, not proof that every plan fails.

The code below builds this pipeline in a synthetic world: an encoder pretrained on frames and then frozen, a learned feature-space predictor, and a CEM planner tested on goal images it has never seen. The closed-loop test will show both progress and misses.