Model-Based Reinforcement Learning › Key Parameters
Michael BrenndoerferJune 12, 20262 min read
World Models HandbookAll chapters Part 2 of 12Inference, Decisions, and Control
Model-Based Reinforcement Learning › Key ParametersOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the Dyna implementation are:
- plan_steps: Number of imagined backups per real environment step. Set it to 0 for plain Q-learning; larger values spend more computation reusing the stored model. The displayed run shows a smaller gain from 10 to 30 than from 0 to 10, but other problems can improve, saturate, or degrade differently. This is the sample-efficiency-versus-compute dial, and, when the model is imperfect, the sample-efficiency-versus-bias dial as well.
- alpha: Learning rate for real and imagined Q-learning updates. It interacts with
plan_steps, but their product is only a rough update-budget heuristic because repeated sampling and Bellman contraction cause saturation. - gamma: Discount factor. Larger values reduce attenuation of distant rewards and weaken Bellman contraction; they do not change graph distance or guarantee that planning helps more.
- eps: Epsilon-greedy exploration rate during data collection, which influences the model's visited-pair coverage. Greedy actions, random tie-breaking, dynamics, and the start state also shape that coverage. Only real interaction adds new pairs in this implementation; planning cannot leave the support already stored in the model.
- n_seeds, n_episodes: Number of random seeds and episodes. Averaging helps, while paired random streams, dispersion bands, and per-seed threshold calculations make the comparison interpretable.