Model-Based Reinforcement Learning › Worked Example: One Trajectory in a Six-by-Six Room

Michael BrenndoerferJune 12, 20262 min read
World Models HandbookAll chapters
Part 2 of 12Inference, Decisions, and Control
Chapter 12Model-Based Reinforcement Learning › Worked Example: One Trajectory in a Six-by-Six Room
Reading section 6 of 11 · Worked Example: One Trajectory in a Six-by-Six Room · 2 minSection 6 of 11 · 2 min

Book contents

World Models Handbook12 parts · 72 chaptersView the whole book →
Active chapter 12 of 72

Section 6 of 11 · 2 min read

Model-Based Reinforcement Learning › Worked Example: One Trajectory in a Six-by-Six RoomOne section · Sections in this chapter6 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's make the Dyna effect concrete before coding it. The room is a 6×66 \times 6 grid. The agent starts in the top-left corner, the goal is in the bottom-right, each nonterminal step costs −1-1 reward, and the transition into the goal pays +10+10 instead. The optimal path is ten moves, for a return of 10−9=1.010 - 9 = 1.0. The geometry matters: the reward sits at the far corner of the state space from the start, so the value information must travel the full diagonal, and any method that propagates value one edge per real visit will pay the full cost of that distance in real interactions.

Now suppose the agent's first episode is an undirected random walk that stumbles into the goal after 60 moves, collecting a return of about −49-49. What does each kind of agent extract from that single trajectory?

The pure Q-learning agent performs 60 update operations, although repeated state-action pairs mean fewer than 60 distinct entries may change. The sample does not guarantee that its second episode will improve.

With plan_steps=30, the Dyna agent performs 30 model backups after each real step, or 1,800 planning backups during a 60-step episode. This implementation samples only state-action pairs that were actually visited. It replays their learned outcomes in a new order; it does not infer untried actions or leave the visited-pair support. Repeated backups can nevertheless propagate reward information backward through the stored graph before another real episode begins.

The honest caveat: this gridworld has a deterministic environment, so the learned model is exact. That is the best case for Dyna, and we chose it deliberately so you can see the mechanism cleanly, with zero model bias contaminating the comparison. Whatever gap appears between the curves is attributable purely to the amplification of honest experience, not to any advantage the model gained by being wrong. The experiment below quantifies exactly how large the advantage is in real interactions.