Intellectual and Cognitive History › A Small Experiment: What a Learned Model Buys You

History makes claims; let's test one of them. Dyna argues that a learned model can extract more updates from each piece of real experience. We test that claim in a small NumPy gridworld by comparing Q-learning with Dyna under the same learning rule and hyperparameters. This is a Dyna experiment, not a miniature DreamerV3: Dyna interleaves replayed model transitions with direct updates, whereas DreamerV3 trains policies through imagined trajectories inside a learned latent model. The loose lineage is their shared use of learned dynamics to turn real experience into additional learning signal. Planning consumes extra random draws, so paired seeds do not produce identical trajectories; averaging twenty independent seeds reduces that sampling noise while leaving planning steps as the systematic treatment difference.

The environment is an obstacle-free $5 \times 5$ grid. The agent starts in the top-left corner and must reach the goal in the bottom-right. Each non-goal step costs $-1$ reward; the transition entering the goal receives $+20$ instead of the step cost and ends the episode. With probability $0.1$ the environment replaces the commanded action with one sampled uniformly from all four actions. The commanded action therefore still executes with probability $0.9 + 0.1/4 = 0.925$. The optimal no-slip episode return is $20 - 7 = 13$, since the shortest path takes eight actions and only the first seven receive $-1$. The executed display below is the canonical record of the twenty-seed outcomes.

First, the environment:

Note the structure of env_step: it is a forward model of a tiny world, written by hand. The agent below does not inspect its code or constants; by experimental convention it receives only the sampled (next_state, reward) outputs and learns a separate empirical model from them. The clipping at the grid edges plays the role of boundary conditions in a physical model. The slip branch is a stochastic disturbance relative to a naive deterministic model, but repeated observations let the empirical model below recover its outcome frequencies.

Next, the agent itself. It is one function with a single switch: planning_steps controls how many Dyna-style simulated updates run between real steps. Setting it to $0$ gives pure model-free Q-learning; setting it to $10$ gives Dyna. Both conditions use $\alpha=0.3$, $\gamma=0.95$, $\epsilon=0.05$, 3,000 real interactions, and a 60-step episode limit. Returns plotted below are undiscounted episode sums even though Q-learning discounts future rewards. A timeout resets the episode but is bootstrapped as a time-limit truncation rather than treated as a terminal transition; goal transitions alone use zero bootstrap.

Three details deserve your attention, because each is a miniature version of a theme from this chapter:

• The model is an empirical one: for each state-action pair it stores a list of observed outcomes and samples from that list during planning. With enough visits, those frequencies converge toward the environment's slip distribution. This is system identification in its crudest form, a topic Part II develops properly. The model cannot invent an outcome absent from its observations, which prevents one kind of optimistic hallucination but leaves it ignorant about unseen state-action pairs.
• Planning updates are identical in form to direct updates. In the code, the planning loop uses the same Q-learning equation but draws the transition from model instead of env_step. The update rule does not encode the source; the caller determines whether a sample is real or simulated.
• The agent never executes a long open-loop plan. Its "plan" is implicit in the value function that planning has improved. At every step it acts epsilon-greedily with respect to current values, then incorporates the newest observation. This resembles the closed-loop humility of receding-horizon control in spirit, but it is not MPC: there is no online trajectory optimization or finite-horizon action sequence. The model shapes the values, the values shape the action, and reality gets a vote at every step.

Now we run the experiment: pure Q-learning versus Dyna with 10 planning steps per real step, averaged over 20 random seeds. This is a closed-loop learning comparison measured in real environment interactions, not an open-loop rollout-accuracy test. The plotted returns come from each agent's online epsilon-greedy behavior policy, including its 0.05 exploration probability; they are not separate greedy-policy evaluations. To compare agents that finish episodes at different times, we interpolate each seed's episode returns onto a common grid spaced every 50 environment steps. Threshold crossings are therefore quantized to that grid. If a run ends midway through an episode at step 3,000, the right-edge record is that time-limit-truncated partial return rather than a completed episode. We report means without uncertainty bands because this is a compact demonstration rather than a statistical benchmark.

The execution output above makes the comparison concrete without duplicating stochastic decimals in prose. For the declared seeds, Dyna learns faster in real-interaction units and reaches the displayed threshold before pure Q-learning. Ten planning steps mean eleven Q-updates per real step, one direct and ten replayed. The planning-depth sweep shows where measured gains occur in this execution; its printed values and bar labels, rather than a frozen prose pattern, are the canonical record. Additional replay cannot compensate for state-action pairs the empirical model has not yet observed, but this experiment does not isolate coverage from exploration, learning rate, grid quantization, or performance-ceiling effects. The no-slip optimum is 13, while slip, persistent epsilon-greedy exploration, and residual value error help explain the lower online returns. Exact returns, threshold crossings, sweep values, and the active NumPy version are recomputed deterministically from EXPERIMENT_SEEDS during conversion. Executable acceptance criteria require Dyna to lead at 500 steps, cross the threshold earlier, and produce a threshold crossing for every plotted sweep condition.

Now the learning curves:

And the planning-depth sweep:

Read these two figures together as a small illustration of the computation-for-experience trade. Simulated transitions accelerate learning in this gridworld under the declared seeds, while the labeled sweep shows how the measured reduction in real interactions varies with planning depth. Differences of one reporting interval should not be read as a general law because the threshold is sampled only every 50 steps. Planning cannot add observations that the model has never received, but the sweep alone does not prove that coverage is the active bottleneck.