Genie and Interactive Generated Environments › Evaluation of Control and Consistency

Evaluating an open-ended generated environment is harder when no reference simulator covers its possible action branches. Where a real environment and fixed action sequence exist, as in our toy, we can measure MSE against the matching reference rollout. We also use behavioral proxies because pixel similarity alone misses control and persistence. This chapter computes six measures:

• Action adherence. Given a requested action, does the model's prediction reflect that action in a way consistent with the environment's mechanics? In our grid world, we can check whether the agent in the predicted frame ends up in the position that the true environment would produce.
• One-step transition error. How close is a single model prediction to the true next frame, when the input is a real frame and a real action? This is the easiest metric to compute and the least informative about interactive quality.
• Multi-step rollout error. How close is a sequence of model-generated frames to the true sequence under the same action sequence? This captures compounding drift.
• Branch separation. How different are rollouts from the same start state under different actions? This captures action responsiveness.
• Revisit consistency. If the agent leaves a state and returns to it, does the model predict the same observation? This captures the model's ability to maintain persistent structure.
• Downstream control. Does a specified planner reach a goal when it selects actions using the model's predictions? This tests the predictor and planner together, not the model in isolation.

No single one of these proxies is sufficient. Action adherence can be inflated by an action-agnostic predictor of a globally common outcome, especially when no-ops dominate. One-step error can be low for a model that ignores actions entirely for the same reason. Multi-step rollout error can be dominated by drift. Branch separation can be produced by noise. Revisit consistency can be satisfied by a model that outputs a constant. The proxies are useful only together, and even together they do not fully characterize playability. They are a practical compromise: cheap to compute, interpretable, and correlated with the properties that matter.

We already computed one-step error, multi-step rollout error, and branch separation. Let us add the other three.

Revisit consistency asks a slightly different question. Suppose the agent moves away from a cell and comes back. Does the model put the agent back in the same cell? We test this with a short sequence that leaves the start position and returns to it. Formally, if the sequence of actions $a_{1:k}$ maps the true state back to the starting position $s_0$, then a revisiting-consistent model should predict an observation close to $o_0$ after feeding its own outputs back k times. The MSE we compute next measures how far the final prediction drifts from the original observation.

Revisit consistency is a minimal test of persistence, not proof of scene memory. Our selected action sequences return the true agent to its start; we compare each model's final recursive prediction with the initial frame. A constant-output model could score well on this comparison while failing action adherence and branch tests, so the measures belong together. The persistence baseline is exactly zero by construction and is not the best interactive model.

Finally, we test downstream control. Give the model a target cell, use it to plan a sequence of actions greedily (each step, pick the action whose predicted next frame is closest to the target), execute those actions in the true environment, and check whether the agent actually reaches the target. Formally, for a start state $s_0$ and target $s^{\star}$, the planner selects at each step the action $a_t$ that minimizes the Manhattan distance between the predicted position $\mathrm{pos}(\hat{o}_{t+1})$ and $s^{\star}$. This is a crude but honest test of whether the model is useful for control, not just for prediction.

The planner is greedy: at each real environment state it rerenders the true observation, scores four one-step model predictions, executes the selected action, and repeats. Thus it does replan after each step, but has no multi-step lookahead and is given privileged access to the true current state. A poor result may reflect model error, the greedy rule, target geometry, or the small trial set; it does not prove a stronger planner would fail. The test asks only whether this particular predictor-plus-controller reaches a target within 20 steps.

The grouped bars compare action adherence and downstream control. In this seeded run, adherence is about 0.57 for the true-action predictor and 0.52 for the calibrated latent-code predictor. Each learned-model controller reaches one of 15 targets. The same greedy rule with exact one-step simulator dynamics reaches 10 of 15. Greedy search and walls still limit that oracle. On these trials, the learned predictors add a substantial failure source beyond the planner.

The corner exposes a local identifiability limit. At $(0,0)$, both "up" and "left" produce no movement. A frame-pair-only inverse model cannot determine which was pressed from that transition. The labeled predictor still receives different action inputs and may output slightly different frames, so the next diagnostic measures its learned prediction gap rather than a proof of perfect ambiguity.

At $(0,0)$, both "up" and "left" are blocked and have the same true next frame. No frame-pair-only inverse model can recover which of those buttons was pressed from that single transition. The supervised true-action predictor can still receive distinct button labels, and a finite fitted model may give slightly different outputs for them; the measured gap is therefore not guaranteed to be zero. The ambiguity is about inferring the cause from the observed pair, not about whether a labeled predictor can distinguish its inputs.

This is a limit on recovering the pressed button from that frame pair, not a claim that two buttons are globally indistinguishable. They can have different effects elsewhere, and a supervised predictor can receive their labels. More subtle data-coverage problems arise when rare state–action interactions are underrepresented. Latent-action discovery cannot infer a distinction that never appears in its observations; evaluation should separate local no-op ambiguity from global action responsiveness.

We can quantify the identifiability gap by comparing the corner with a free cell where the two actions have different effects.

This heatmap makes two distinct ambiguities visible. The numeric code labels are arbitrary and could be permuted without changing the clustering. More importantly, the mixed row is not an arbitrary relabeling of one button: blocked actions from several buttons produce the same zero displacement, so their intent cannot be recovered from the observed frame pair. The clustering uses displacements computed from the two frames, not the button labels. Evaluation must distinguish arbitrary code identity from the information the observation actually contains.

The heatmap also makes the no-op absorption visible. If a row has counts spread across several true actions, that latent code is not cleanly aligned with a single direction. That can happen because the code is absorbing no-op transitions, or because the clustering has merged two similar directions. In a real latent-action model, this kind of analysis is harder because the codes are not directly interpretable as directions, but the same principle applies. You want to know which codes are well used, which are collapsed, and which correspond to mixtures of changes. The confusion matrix is a simple tool for asking those questions in a setting where you happen to have the true labels.