Model-Based Reinforcement Learning

Imagine a robot arm that must learn to insert a peg into a hole. A model-free reinforcement learning agent treats the arm as a black box: it tries actions, observes outcomes, and slowly accumulates value estimates through trial and error. On physical hardware this is painful. Every bit of learning costs real time, real wear on the motors, and real risk of damage. If a policy requires a hundred thousand training interactions, and each interaction takes a second of wall-clock time and a little life off the gearboxes, the arithmetic of deployment becomes unforgiving. A model-based agent takes a different route: it first learns how the world responds to its actions, then uses that learned model to rehearse. It can imagine thousands of peg insertions in the privacy of its own simulator, extract what it needs, and only occasionally return to reality to check its work. The expensive resource, real interaction, is spent sparingly; the cheap resource, computation inside the learned model, is spent liberally.

This is the promise of model-based reinforcement learning (MBRL): use a learned model to obtain more decision-making signal from each real interaction. Whether that promise is realized depends on the task, the data, and how the model is consumed. Model errors do not enter every decision uniformly; optimization can favor trajectories where the errors look advantageous. This chapter follows both sides of that bargain: how dynamics models support planning, policy optimization, and value learning, and how model bias, compounding rollout error, and model exploitation constrain those uses.

Throughout, keep two questions separate, because conflating them causes most MBRL confusion:

• What does the model predict? States, observations, rewards, sometimes whole futures.
• How does the agent use those predictions? To plan at decision time, to train a policy, or to train a value function.

The first question is about the model itself: its inputs, outputs, accuracy, and uncertainty. The second is about the decision machinery wrapped around it: whether predictions are consumed one step at a time, unrolled open-loop, or differentiated through. A given model can be paired with any of these consumers, and the same model can be excellent for one use and poor for another. Keeping the questions separate lets you diagnose failures precisely: when an MBRL system underperforms, the fault may lie in the model, in the consumer, or, most often and most subtly, in the interaction between the two.

Building on the MDP formalism from Part II: Inference, Decisions, and Control, the agent faces a Markov decision process with transition kernel $p(s_{t+1} \mid s_t, a_t)$ and reward function $r(s_t, a_t)$. A model-free decision method learns a policy $\pi_\theta$ or value function $Q_\theta$ from experience without querying an explicit $p$ or $r$ during decision learning. An implementation may still train predictive auxiliary tasks; the distinction is whether a transition-and-reward model supplies the policy, value, or planning update. Model-based methods insert that intermediate object:

Read the diagram left to right as a data flow, not a one-time pipeline: real experience refines the model, and the model feeds decision machinery whose behavior determines what data is collected next. A learned model is an explicit, reusable summary of experience. It can be queried for actions not taken in a recorded transition when its representation and data support such generalization, and it can generate additional optimization inputs. Those synthetic transitions are not new independent evidence. Model-free policies, values, representations, and replay data can also be reused; the practical distinction is that a model exposes a transition-and-reward query interface.

Chapters 3 and 4 of this part already showed you both ends of this pipeline: system identification taught you how to fit a dynamics model from data, and classical optimal control taught you how to act with a known model. Model-based RL couples these activities online. Adaptive and dual control also study this controller-model-data feedback; MBRL brings the same problem into reinforcement-learning systems with learned, often high-capacity models. As the controller changes, its state-action distribution can change, so validation must track the distribution on which the model will actually be used.

The components of a model-based RL system are:

• Dataset $\mathcal{D}$ of real transitions $(s_t, a_t, r_t, s_{t+1})$ collected by earlier behavior. Its distribution often shifts as the policy changes, although it can be stationary under a fixed behavior policy.
• World model $\hat{p}_\phi$ and reward model $\hat{r}_\psi$, fit to $\mathcal{D}$ by likelihood or regression. MBRL often augments ordinary system-identification losses with uncertainty models, multi-step objectives, online updates, or control-aware weighting.
• Decision machinery, one of the three mechanisms described below: a planner, a policy, or a value function. This is the consumer whose needs the model must ultimately serve.
• Data-collection strategy, which decides what real experience to gather next, often by exploring where the model is uncertain or where the current policy performs poorly. This component closes the loop between the agent and the dataset. Model uncertainty can support model-aware exploration, although model-based and model-free systems can also use the same behavior policy.

Here is the trap that separates people who have run MBRL experiments from people who have only read about them. You fit the model to minimize one-step prediction error on the data you collected, exactly as system identification recommends. The model gets impressively accurate. You then let the agent plan with it, and the agent behaves worse than a model-free baseline. What went wrong?

The first time you see this, it feels like a contradiction. The model is measurably good; the decisions built on it are measurably bad. The resolution is that "good" was measured under the wrong conditions.

The mismatch has two root causes. First, distribution shift: as the policy improves, it visits states the dataset never contained, and the model extrapolates there. A neural network fit on random-walk data has no idea what the world looks like along the precise, purposeful trajectories a trained policy would take, and nothing in its training loss asked it to. Second, compounding: planners use the model recursively, feeding its own predictions back in as inputs, so one-step errors accumulate over the rollout horizon. Even a model whose one-step error is negligible on-distribution can drift badly over a twenty-step rollout, because step twenty's input is nineteen of its own guesses. Both deserve their own treatment, and we return to them in detail below.

There is also a subtler conceptual point, foreshadowed in the world-model qualification test from Part I: The World-Model Idea: open-loop prediction quality and closed-loop decision usefulness are different properties. A model can pass every prediction benchmark you throw at it and still produce bad policies, because policies live in a different part of the distribution than prediction benchmarks do. A benchmark scores the model where the data was; a policy uses the model where the optimizer steered. We will develop evaluation criteria for this in Part XI: Evaluation and Understanding of the handbook; for now, hold onto the distinction as a design principle. Whenever you train a model inside a decision system, ask not only "how accurate is it?" but "accurate where, and under what distribution of queries?"

A useful taxonomy groups model use into three mechanisms: decision-time planning, policy optimization through imagined trajectories, and value learning from imagined experience. Many systems combine them, and some auxiliary-model or representation-learning methods do not fit the categories cleanly.

The most direct use: at every decision point, optimize an action sequence through the model, execute a prefix, and repeat. This is model predictive control (MPC), which you met in classical form in the optimal control chapter; the only change is that the dynamics come from $\hat{p}_\phi$ instead of a hand-derived ODE:

This objective is risk-neutral: it scores a candidate by expected return under the model. Risk-sensitive or pessimistic MPC replaces that expectation with another functional. After executing $a_t^*$, the agent observes the next state and replans. This feedback usually limits exposure to open-loop drift, but it does not guarantee robustness to a biased model or make the inner optimization easy.

For continuous actions, sampling-based optimizers such as random shooting, CEM, and MPPI are common, alongside gradient methods and trajectory optimizers such as iLQR. Stochastic rollouts can still provide pathwise gradients through reparameterization or score-function estimates; sampling is attractive when such gradients are unavailable, unstable, or not worth their cost. PETS (Chua et al., 2018) paired CEM with a probabilistic neural-network ensemble and reported strong sample efficiency on its continuous-control benchmarks. Tree search is important for many discrete-action problems; MuZero searches in a learned latent model, as discussed in Part VIII: Decision-Centric Research Lineages.

Planning has a distinctive profile:

• No separate policy training is required. Plain online MPC selects actions directly, although learned proposal policies and planner warm starts are common. With learned nonlinear dynamics, the inner optimization is often nonconvex; linear MPC with convex costs and constraints can instead be convex. Numerical behavior and closed-loop stability depend on the full formulation and solver.
• Decision-time compute. Sampling budgets range widely with the optimizer, horizon, model, and hardware. This can be difficult under hard real-time constraints, although learned-model MPC can run at high rates in suitably engineered systems.
• Short horizons are natural. Because reality corrects the state estimate after each executed action, MPC can succeed with horizons where open-loop planning would fail. The model is exposed to at most $H$ consecutive predicted transitions even when the task lasts much longer.

The second mechanism treats the model as a differentiable simulator and backpropagates value through it to train a policy directly. For the expression below, assume a deterministic differentiable transition model, a deterministic differentiable policy $a_k=\pi_\theta(s_k)$, and a differentiable modeled reward. Then $\hat{s}_{t+k+1} = f_\phi(\hat{s}_{t+k}, \pi_\theta(\hat{s}_{t+k}))$, the finite-horizon return is differentiable in $\theta$, and you can compute gradients through the imagined trajectory. A stochastic policy instead needs reparameterization or another valid gradient estimator.

where $a_k=\pi_\theta(s_k)$ and the state sensitivity obeys the corresponding recursion through $f_\phi$. These sensitivities multiply repeated model Jacobians, so they can vanish or explode with horizon. PILCO (Deisenroth and Rasmussen, 2011) used Gaussian-process dynamics, approximate analytic moment propagation, and analytic policy gradients, solving its reported control tasks with very little interaction. Its posterior uncertainty is meaningful under its modeling assumptions, but is not automatically calibrated. Gradient-based and sampling-based optimizers can both exploit model error; the result depends on their search pressure, regularization, and the geometry of the learned model.

The Dreamer family learns latent dynamics and trains actor and value functions from imagined latent rollouts. The original Dreamer propagates actor gradients through imagined dynamics; estimator details vary across versions. A latent state can omit task-irrelevant observation detail, but it is not necessarily lower-dimensional and does not shorten the temporal chain by itself. Part III: Representing Agents and Worlds develops the representation machinery.

The third mechanism is the oldest and, in some ways, the most durable: use the model to generate synthetic transitions, and feed them to an otherwise ordinary model-free learner. The value backup

has the same algebra for a real or modeled transition. In practice, model bias and the synthetic-data distribution often require different rollout lengths, sampling ratios, weighting, or pessimism. The model supplies additional optimization inputs and can branch from real states into continuations not present in the replay buffer; it does not supply additional independent evidence.

This is the Dyna family, which gets its own section below. MBPO (Janner et al., 2019) is a modern relative that trains soft actor-critic on short model rollouts branched from real replay states and reported strong sample efficiency on its benchmarks.

A useful refinement is model-based value expansion (Feinberg et al., 2018): unroll $H$ modeled steps from a real state, sum predicted rewards, and bootstrap from a terminal value estimate. Increasing $H$ reduces reliance on the terminal bootstrap, but it can increase sampling variance and exposure to accumulated model bias; the balance depends on the estimator, model, and task. The rollout rewards are model predictions, not additional real reward observations.

The three mechanisms are compared in Table model-uses:

These are not rivals so much as a menu. PETS combines a learned ensemble model with planning. Dreamer combines value learning with gradients through the model. MBPO combines Dyna-style imagined data with off-policy value learning. The table's failure column is a diagnostic aid: ask how the model is consumed, then ask how the method limits harmful distribution shift and model exploitation.

This section covers the central tradeoff of the chapter. Models buy sample efficiency, and they charge for it in bias. The two subsections that follow give each side its due, first the mechanisms by which models reduce the demand for real data, then the quantitative ways in which model error enters and accumulates. By the end, you should be able to identify the controlling tradeoffs and design evaluations that reveal whether a model-based system helps on a particular problem.

It is worth being precise about where the efficiency comes from, because "the model lets you imagine" is only part of the story. There are at least four distinct sources, and they interact:

• Compression. A suitable parametric model can share structure across states rather than learn each tabular state-action pair independently. This statistical advantage depends on whether the hypothesis class captures structure that actually transfers.
• Re-use and re-weighting. Both model-free replay and model-based training can revisit real transitions many times. A model adds structured branching: it can propose continuations and reweight where computation is spent.
• Off-policy imagination. A generalized model can evaluate actions absent from a particular trajectory, but reliable counterfactual prediction still requires action coverage or defensible structural assumptions. State coverage alone is not enough.
• Auxiliary predictive signal. Predicting observations or dynamics provides dense self-supervised targets even when reward is sparse. A learned reward model does not by itself densify sparse reward labels; that requires shaping, auxiliary objectives, or other added structure.

These potential benefits do not require long-horizon rollouts. A system may use a model only for short branching, auxiliary prediction, or one-step backups.

Suppose the true and learned transition maps start from the same state, so $e_0=0$, and are evaluated under the same actions. Assume the learned map has uniform one-step error at most $\varepsilon$ on the relevant region and the true transition map is $L$-Lipschitz in state. If $e_h$ is the state error after $h$ modeled transitions, then

An error injected $k$ transitions before the end is multiplied by at most $L^k$. The regimes differ: the bound stays finite as $H$ grows when $L<1$, grows linearly when $L=1$, and grows exponentially only when $L>1$.

The bound is easiest to internalize as a curve. Fix a one-step error of $\varepsilon = 0.01$, a deliberately small value that would look excellent on any one-step benchmark, and plot the worst-case rollout error against horizon for a few values of the Lipschitz constant $L$:

The panels support a narrower and more useful conclusion. On these paired trajectories, the two models have one-step errors of the same order, with no consistent winner over the first five steps. By step 30 the neural model has the lower rollout error. One-step ranking therefore does not determine long-horizon ranking, and recursive use must be evaluated directly. A trained policy would induce another test distribution, so these random-action results should not be generalized beyond the displayed experiment.

Compounding error degrades rollouts passively. A nastier failure mode is active: the decision machinery seeks out the model's errors.

The interaction is between optimization and distribution shift. A planner may steer toward unfamiliar states or actions where the model is unreliable; those regions need not be the true high-reward regions. If optimistic errors dominate there, additional planning can worsen real performance even while imagined return improves.

One common defense is model uncertainty. Bootstrap ensembles often provide a useful proxy for epistemic uncertainty. Disagreement can expose sensitivity to finite data, but it can miss shared model-class bias and is not automatically calibrated.

To see both the value and the limitation, fit quadratic models to bootstrap resamples of noisy sine data. The members share the same misspecified feature class, so their spread measures resampling sensitivity within that class, not total predictive uncertainty.

The widening band outside $[-1,1]$ is a useful warning about extrapolation. The dashed truth also exposes the limitation: a thin band is not a certificate of correctness when all members share a misspecified quadratic class. An uncertainty penalty or rollout truncation can still be useful, but it should be calibrated and checked against control-relevant held-out data.

PETS averages over ensemble rollouts in its planner; many other methods add an uncertainty penalty directly to the objective. Other common defenses include:

• Short rollouts and branching. Generate imagined data only a few steps from real states, then bootstrap with the value function. This bounds compounding by construction: every imagined trajectory starts from ground truth and stays short. This is MBPO's core insight and, as we saw, it is also implicit in MPC's re-planning.
• Pessimism. One common implementation subtracts an uncertainty estimate from imagined reward so that unfamiliar, possibly-wrong regions look bad rather than promising. Other implementations use lower-confidence values, robust transition sets, constraints, or worst-case objectives. These constructions connect to the same pessimism principle used in offline RL.
• Model trust diagnostics. Before letting the agent act on a model, test it the way the qualification test from Part I: The World-Model Idea suggested: not only on held-out data, but on the distribution the policy will induce, and at the rollout lengths the decision machinery will use. A model validated only on its training distribution has not been validated at all, as far as the decision machinery is concerned.

Planner error and model error are distinct. A precise optimizer can expose optimistic model errors, while an approximate optimizer can simply be suboptimal. Robust objectives, constraints, uncertainty penalties, and real-environment validation are more principled controls than assuming that fewer optimizer iterations always make an uncertain model safer.

Richard Sutton presented an early version of Dyna at ICML in 1990 and developed the framework further in 1991. Dyna remains a clear template for algorithms that interleave real experience, model learning, and planning updates. It is an important lineage for modern MBRL, not an exhaustive description of every model-based method.

Each cycle of interaction does three things:

1. Act in the real environment: choose $a_t$ from the current policy, observe $(r_t, s_{t+1})$.
2. Learn the model: update $\hat{p}_\phi$ and $\hat{r}_\psi$ with the new transition.
3. Learn the value, twice: apply one real Q-learning update from the observed transition, then apply $n$ additional planning updates, each drawn from the model: sample a previously visited $(s, a)$, query the model for $(\hat{r}, \hat{s}')$, and apply a backup as if it had really happened.

The $n$ planning updates are what distinguishes Dyna from plain Q-learning. Each one propagates value information backward through the state space without touching the environment. The imagined backups are interleaved with the real ones, so real updates continue to provide ground-truth corrections while modeled updates propagate information more deeply. Those real corrections do not guarantee protection from bias when inaccurate modeled backups dominate. Notice also the division of labor: the model absorbs the raw statistics of the environment, while the value function absorbs the long-horizon consequences. Neither has to do the other's job.

The tabular Q-learning backup is local: a transition moves value information one edge from $s'$ to $s$. Dyna repeats such backups from stored model entries between real interactions, so reward information can propagate through already visited state-action pairs without revisiting them in the environment. The size of the gain depends on exploration, planning samples, hyperparameters, and model accuracy.

Two assumptions carry this argument:

• The Markov assumption from Part II: Inference, Decisions, and Control, Chapter 1 must hold for the model's inputs. If the environment is partially observable and you model raw observations, the model is predicting from an insufficient statistic, and the imagined backups will be systematically wrong. The model may fit the data well in the one-step sense while still being useless for planning, because no function of the observation alone can capture the hidden state the future depends on. Belief-state modeling from Chapter 2 of this part is the fix, and latent-state models in Part III: Representing Agents and Worlds are the learned version of the same idea.
• The model must be right where the planning distribution samples it. In tabular Dyna with a deterministic environment, the model is exact after one visit per pair, so imagined data is as good as real data. The moment the model is approximate, imagined backups inject model bias directly into $Q$, and the value function becomes a compounding of model error over the effective planning depth. Dyna amplifies a good model and a bad one. This is the sample-efficiency-versus-bias tradeoff in its purest form: the planning multiplier $n$ scales both the honest signal and the bias.

Dyna is closely related to prioritized sweeping (Moore and Atkeson, 1993), which spends planning updates where the value estimate changed most and is an ancestor of prioritized replay in model-free deep RL. And the modern branch-based methods are best read as Dyna with a smarter sampling rule: instead of replaying visited pairs, sample real states from a replay buffer, branch short imagined continuations from them, and train a deep value function on the mixture. Dreamer's actor and critic train on imagined latent trajectories, while its world model is still fit to real replay. Read that way, three decades of MBRL research look like a long refinement of a single question: from which states, and over what horizons, should imagined experience be generated?

Let's make the Dyna effect concrete before coding it. The room is a $6 \times 6$ grid. The agent starts in the top-left corner, the goal is in the bottom-right, each nonterminal step costs $-1$ reward, and the transition into the goal pays $+10$ instead. The optimal path is ten moves, for a return of $10 - 9 = 1.0$. The geometry matters: the reward sits at the far corner of the state space from the start, so the value information must travel the full diagonal, and any method that propagates value one edge per real visit will pay the full cost of that distance in real interactions.

Now suppose the agent's first episode is an undirected random walk that stumbles into the goal after 60 moves, collecting a return of about $-49$. What does each kind of agent extract from that single trajectory?

The pure Q-learning agent performs 60 update operations, although repeated state-action pairs mean fewer than 60 distinct entries may change. The sample does not guarantee that its second episode will improve.

With plan_steps=30, the Dyna agent performs 30 model backups after each real step, or 1,800 planning backups during a 60-step episode. This implementation samples only state-action pairs that were actually visited. It replays their learned outcomes in a new order; it does not infer untried actions or leave the visited-pair support. Repeated backups can nevertheless propagate reward information backward through the stored graph before another real episode begins.

The honest caveat: this gridworld has a deterministic environment, so the learned model is exact. That is the best case for Dyna, and we chose it deliberately so you can see the mechanism cleanly, with zero model bias contaminating the comparison. Whatever gap appears between the curves is attributable purely to the amplification of honest experience, not to any advantage the model gained by being wrong. The experiment below quantifies exactly how large the advantage is in real interactions.

We now build the worked example in full. The plan: implement the gridworld and a tabular Dyna agent, run three variants (no planning, 10 planning steps, 30 planning steps) across many random seeds, and plot episode return against real environment interactions, which is the resource we care about. Plotting against episodes instead would flatter nothing and show nothing: an episode is a variable-length bundle of interactions, and the whole point of model-based methods is their exchange rate between computation and reality.

First, the environment. It is deterministic, with bounded grid movement and a terminal goal state. The clipping at the walls means attempted moves off the grid result in staying in place, which gives the agent a small but real amount of transition structure to learn about the boundary.

In this exact deterministic tabular setting, the two planning variants reach the displayed threshold in roughly one-third of Q-learning's real interactions. The difference between 10 and 30 planning updates is smaller than the difference between either Dyna variant and no planning. This is one experiment, not a monotonic law: with an approximate model, more planning can reinforce bias, while with an exact model and a compute-free objective, no theorem here requires a finite optimal planning budget.

You now have a complete Dyna-style MBRL system: model learning, imagined backups, and explicit accounting of real interactions. Other model-based methods may instead plan directly or differentiate through a model without this value-learning loop.

The key parameters for the Dyna implementation are:

• plan_steps: Number of imagined backups per real environment step. Set it to 0 for plain Q-learning; larger values spend more computation reusing the stored model. The displayed run shows a smaller gain from 10 to 30 than from 0 to 10, but other problems can improve, saturate, or degrade differently. This is the sample-efficiency-versus-compute dial, and, when the model is imperfect, the sample-efficiency-versus-bias dial as well.
• alpha: Learning rate for real and imagined Q-learning updates. It interacts with plan_steps, but their product is only a rough update-budget heuristic because repeated sampling and Bellman contraction cause saturation.
• gamma: Discount factor. Larger values reduce attenuation of distant rewards and weaken Bellman contraction; they do not change graph distance or guarantee that planning helps more.
• eps: Epsilon-greedy exploration rate during data collection, which influences the model's visited-pair coverage. Greedy actions, random tie-breaking, dynamics, and the start state also shape that coverage. Only real interaction adds new pairs in this implementation; planning cannot leave the support already stored in the model.
• n_seeds, n_episodes: Number of random seeds and episodes. Averaging helps, while paired random streams, dispersion bands, and per-seed threshold calculations make the comparison interpretable.

The risk of model error is structural when predictive models are learned from finite data, even though a finite deterministic tabular model can become exact on each observed pair. Open-loop state error can remain bounded for contractive dynamics, grow linearly at $L=1$, or grow exponentially for $L>1$. Discounted value error has its own scaling and depends on whether error is measured in rewards, transitions, or Bellman backups. Common defenses include uncertainty estimates, short rollouts, pessimism, constraints, and validation on the decision distribution, but no one defense is universal. Model-based methods also trade data for computation. Under partial observation, the model needs a sufficient statistic such as a belief or recurrent state; one-step validation on aliased observations can otherwise look good while planning fails.

Model-based RL has produced influential sample-efficient systems. PILCO reported solving a physical cart-pole task with 17.5 seconds of interaction. PETS, MBPO, and Dreamer extended different model-use mechanisms to neural models and high-dimensional observations. DreamerV3 learned Minecraft diamond collection without human data by learning behavior from imagined rollouts while still collecting and acting on real environment experience. Dyna remains one important lineage among the decision-centric methods in Part VIII: Decision-Centric Research Lineages and the planning agents of Part VII: Planning and Agency.

For you as a practitioner, the chapter leaves you with a design checklist rather than a single recipe:

• Decide how the model is used, using planning, policy optimization, and imagined value learning as a practical taxonomy rather than an exhaustive partition.
• Keep imagined rollouts within the model's validated horizon. When the consumer uses a terminal value function, let that value carry the burden beyond the rollout.
• Use an uncertainty-aware diagnostic or objective when model uncertainty is material and can be estimated; an exact known model does not need a proxy for uncertainty.
• Evaluate the model on the decision distribution, not just held-out one-step accuracy, remembering the qualification test from Part I: The World-Model Idea.
• Report and monitor the real-interaction budget, the number that model-based RL exists to shrink.

• Model-based RL inserts a dynamics and reward model between experience and decisions. When data and representation support the query, that model can evaluate alternative actions and generate synthetic optimization inputs.
• A useful taxonomy asks whether the model supports decision-time planning, policy optimization through imagination, or value learning from synthetic transitions; many algorithms combine these uses.
• Under the stated same-start, uniform-error, and Lipschitz assumptions, the worst-case open-loop state-error bound is finite, linear, or exponential according to transition sensitivity. Discounted performance bounds must likewise state the error metric and assumptions.
• Model exploitation occurs when optimization favors optimistic model errors. Ensembles, short branched rollouts, pessimism, constraints, and decision-distribution validation are common defenses, none of which certifies correctness alone.
• Dyna interleaves real updates, model learning, and backups from visited model entries. Direct planners and differentiable policy-search methods need not follow this loop.
• In the deterministic gridworld experiment, both Dyna variants reached a smoothed return of $-2$ in about one-third of Q-learning's mean real interactions across seeds. That result is specific to the implementation and hyperparameters shown.

The next part of the book turns to the representation question this chapter deliberately set aside: what should the model predict over? Raw observations are often an awkward target for control-relevant prediction, and the road to latent, memory-equipped, object-centric world models starts with a closer look at where observations come from.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about model-based reinforcement learning.