Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Model-Based Reinforcement Learning › Dyna: Integrating Real and Imagined Experience

Michael BrenndoerferJune 12, 20264 min read

World Models HandbookAll chapters

Part 2 of 12Inference, Decisions, and Control

Chapter 12Model-Based Reinforcement Learning › Dyna: Integrating Real and Imagined Experience

Reading section 5 of 11 · Dyna: Integrating Real and Imagined Experience · 4 minSection 5 of 11 · 4 min

Section 5 of 11 · 4 min read

Model-Based Reinforcement Learning › Dyna: Integrating Real and Imagined ExperienceOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Richard Sutton presented an early version of Dyna at ICML in 1990 and developed the framework further in 1991. Dyna remains a clear template for algorithms that interleave real experience, model learning, and planning updates. It is an important lineage for modern MBRL, not an exhaustive description of every model-based method.

The AlgorithmLink Copied

Each cycle of interaction does three things:

Act in the real environment: choose $a_t$ from the current policy, observe $(r_t, s_{t+1})$ .
Learn the model: update $\hat{p}_\phi$ and $\hat{r}_\psi$ with the new transition.
Learn the value, twice: apply one real Q-learning update from the observed transition, then apply $n$ additional planning updates, each drawn from the model: sample a previously visited $(s, a)$ , query the model for $(\hat{r}, \hat{s}')$ , and apply a backup as if it had really happened.

In[2]:

Code

## Dyna's inner loop, in pseudocode
for each real step:
    a, s2, r, done = act_and_observe(s)
    model[(s, a)] = (s2, r, done)       # learn the model
    q_update(s, a, r, s2, done)         # real backup
    for _ in range(n_planning):
        ps, pa = sample_visited_pair()
        ps2, pr, pdone = model[(ps, pa)]  # imagined transition
        q_update(ps, pa, pr, ps2, pdone)  # imagined backup
    s = s2                              # advance the real state

The $n$ planning updates are what distinguishes Dyna from plain Q-learning. Each one propagates value information backward through the state space without touching the environment. The imagined backups are interleaved with the real ones, so real updates continue to provide ground-truth corrections while modeled updates propagate information more deeply. Those real corrections do not guarantee protection from bias when inaccurate modeled backups dominate. Notice also the division of labor: the model absorbs the raw statistics of the environment, while the value function absorbs the long-horizon consequences. Neither has to do the other's job.

Why It Works and When It FailsLink Copied

The tabular Q-learning backup is local: a transition moves value information one edge from $s'$ to $s$ . Dyna repeats such backups from stored model entries between real interactions, so reward information can propagate through already visited state-action pairs without revisiting them in the environment. The size of the gain depends on exploration, planning samples, hyperparameters, and model accuracy.

Two assumptions carry this argument:

The Markov assumption from Part II: Inference, Decisions, and Control, Chapter 1 must hold for the model's inputs. If the environment is partially observable and you model raw observations, the model is predicting from an insufficient statistic, and the imagined backups will be systematically wrong. The model may fit the data well in the one-step sense while still being useless for planning, because no function of the observation alone can capture the hidden state the future depends on. Belief-state modeling from Chapter 2 of this part is the fix, and latent-state models in Part III: Representing Agents and Worlds are the learned version of the same idea.
The model must be right where the planning distribution samples it. In tabular Dyna with a deterministic environment, the model is exact after one visit per pair, so imagined data is as good as real data. The moment the model is approximate, imagined backups inject model bias directly into $Q$ , and the value function becomes a compounding of model error over the effective planning depth. Dyna amplifies a good model and a bad one. This is the sample-efficiency-versus-bias tradeoff in its purest form: the planning multiplier $n$ scales both the honest signal and the bias.

Dyna is closely related to prioritized sweeping (Moore and Atkeson, 1993), which spends planning updates where the value estimate changed most and is an ancestor of prioritized replay in model-free deep RL. And the modern branch-based methods are best read as Dyna with a smarter sampling rule: instead of replaying visited pairs, sample real states from a replay buffer, branch short imagined continuations from them, and train a deep value function on the mixture. Dreamer's actor and critic train on imagined latent trajectories, while its world model is still fit to real replay. Read that way, three decades of MBRL research look like a long refinement of a single question: from which states, and over what horizons, should imagined experience be generated?