Games and Learned Game Engines › Agents Trained Inside Generated Games

A learned model can supply imagined trajectories for policy updates or candidate-action search, but the model itself first needs data. Dreamer continues to collect environment experience while learning from imagined trajectories; MuZero learns dynamics useful for search; Ha and Schmidhuber tested a dream-trained controller in VizDoom. GameNGen's DOOM generation is primarily a playable-model demonstration, not evidence that an agent trained in it transferred to the reference game. Computational and data advantages depend on the environment, hardware, model, and evaluation protocol.

A model used for decisions needs at least three properties under the intended task distribution.

1. Decision sufficiency. It should retain information needed to rank actions for the objective, even when visual reconstruction is not required.
2. Rollout stability over the planning horizon. The drift we just measured must be small enough that short plans remain accurate.
3. Reward realism. The predicted reward must be aligned with the true reward on states the planner tends to visit.

These conditions differ from visual playability, rather than forming a guaranteed sufficient test. A model can be useful for one planner while looking poor, and a rendered world can look persuasive while giving wrong rewards. Even when the three conditions appear to hold on a test set, distribution shift under optimization can expose further errors.

The literature separates two styles of use.

• Policy learning in imagination. The agent updates an actor and critic from modeled trajectories, as Dreamer does. The learned policy can exploit errors in the world model; actual training also needs environment data for that model.
• Planning in the model. A planner evaluates candidate futures and chooses an action, as in MuZero's MCTS or the simple MPC below. Replanning from a new observation limits the length of the executed open-loop plan but does not make the first decision immune to errors in simulated futures.

Both styles can exploit errors. An imagined policy is trained against model predictions; a planner can choose a bad first action because an inaccurate later reward made its candidate sequence look attractive. Receding-horizon search then observes the new real state before choosing again, while an actor does not necessarily search anew at each decision. Search also has a per-decision cost that depends on its budget. The example below demonstrates random-shooting MPC, not a general superiority claim for planning.

Optimization can favor model-predicted outcomes that do not occur in the reference environment. For the grid world, separate the true reward rule from the network's reward-head prediction. The environment returns

$$r_t=-\lVert s_{t+1}-g\rVert_1,$$

Here $s_{t+1}$ is the true post-action agent coordinate, $g$ is the true goal coordinate, and $\lVert\cdot\rVert_1$ is Manhattan distance. The learned model instead predicts

$$\hat r_t=R_\theta(o_t,a_t),$$

where $R_\theta$ is the separate scalar reward head after the MLP trunk. It is not computed by decoding the predicted next observation or measuring its distance to a decoded goal. In particular, a one-cell observation-prediction error does not imply a one-unit error in $\hat r_t$. The planner below sums these direct reward-head outputs. Model exploitation remains possible if a candidate action sequence receives optimistic reward predictions, but this code does not demonstrate that it occurs.

The distinction between the two heads matters diagnostically: observation drift and reward error need separate measurements. A visually wrong rollout might still rank actions well, while a visually plausible one might misrank them. Repeated optimization can search for optimistic predictions, but whether that happens here must be measured against true returns.

Common responses to this risk change the model, planner, or objective:

• Ensembles and uncertainty penalties. Model disagreement can signal poorly supported actions, although disagreement is not a perfect error estimate. PILCO, PETS, Ensembles, and MBPO compares related approaches.
• Short imagination horizons. Dreamer imagines finite rollouts and uses a value model to estimate returns beyond them. Shorter rollouts limit how far the learned dynamics are applied directly, although accumulated model error is not the paper's stated reason for this design.
• Real-data correction. New reference-environment transitions can reveal and repair model errors in visited regions, at an interaction cost.
• Conservatism. MOPO subtracts a model-uncertainty penalty from imagined rewards; MOReL routes unsupported state-actions to a pessimistic unknown state. Part VIII's planned chapter on offline and conservative model-based RL covers this family.

Each mitigation changes what the agent is allowed to optimize or what evidence it receives. None guarantees transfer for an arbitrary learned model. Evaluate the chosen policy in the reference environment, not solely against its training model.

At each environment step, random-shooting MPC samples action sequences of length $H$, rolls them out inside the learned model, and selects the sequence with the highest sum of predicted reward-head outputs. It executes only that sequence's first action, observes the real next state, and plans again. Re-anchoring prevents errors in an unexecuted trajectory from becoming the next observed state, but errors at any simulated depth can still affect which first action wins the search.