Intellectual and Cognitive History › Key Parameters

The key parameters for the Dyna experiment are:

• alpha = 0.3: Q-learning step size for both direct and planning updates.
• gamma = 0.95: Discount factor. The conventional horizon scale $1/(1-\gamma)=20$ is roughly two and a half times the grid's eight-action shortest path when every commanded move succeeds, and the goal signal retains a factor of $0.95^8 \approx 0.66$ along that slip-free path.
• eps = 0.05: Exploration probability of the online epsilon-greedy behavior policy that also generates the plotted returns.
• planning_steps = 0 or 10: Simulated updates between real steps in the main comparison; the sweep uses 0, 1, 5, 10, 50.
• n_steps = 3000: Real environment interactions per run.
• timeout = 60: Maximum episode length before a time-limit reset. Timeouts retain a bootstrap value; only the goal is terminal.
• SLIP = 0.1: Probability of replacing the command with an action drawn uniformly from all four actions, leaving the commanded action's total execution probability at 0.925.
• GRID = 5, START = (0, 0), GOAL = (4, 4): Environment geometry behind the eight-action shortest path.
• THRESHOLD = 7.0, STEP_GRID = 100, 150, ..., 2950: Reporting threshold and its 50-step measurement resolution.
• N_SEEDS = 20: Independent seeds averaged into each curve. A formal comparison would also report confidence intervals or other uncertainty estimates.

The four assumptions named in the opening have now met their historical tests. Prediction error organized Helmholtzian and predictive-processing accounts, but only under an inference model. Compact state powered cognitive maps and control, but only when the abstraction retained what the task needed. Matching sensory statistics enabled modern generators, but did not by itself supply interventions or decisions. Simulated experience accelerated Dyna in the experiment above, but only over transitions already represented by the learned model. The rest of the book develops these conditions rather than treating any of the four assumptions as universally valid.