Intellectual and Cognitive History

Watch a goalkeeper dive to stop a penalty kick. From the instant of contact, a firmly struck ball can cross the eleven meters from the penalty mark to the goal in roughly half a second. Flight time depends on shot speed: at roughly 22 to 32 meters per second, the simple estimate $t = 11/v$ gives about 0.34 to 0.50 seconds. A goalkeeper cannot wait for the full flight, interpret it, choose a direction, and then move across most of the goal. Instead, the goalkeeper uses early visual cues to anticipate where the ball will be, a behavior studied directly in in-situ penalty experiments. This is usefully modeled as a short-horizon internal simulation of a physical system, run forward from partial and noisy observations. The goalkeeper must commit the body's gross trajectory before takeoff, while vision can still guide smaller adjustments of the limbs during the dive.

Notice how much is packed into this single everyday act. The goalkeeper's brain maintains an estimate of the ball's position and velocity from delayed, noisy visual evidence. It extrapolates that state through learned regularities of ball flight, while coping imperfectly with aerodynamic effects such as drag and spin, and evaluates candidate interception points against the body's reach, momentum, and the geometry of the goal. Because the flight is so short, takeoff largely fixes the body's center-of-mass trajectory even though limb configuration can still be adjusted. Unlike the full receding-horizon controllers we meet later, the goalkeeper commits the body's gross trajectory early and adjusts only locally afterward. Its engineering counterparts are state estimation, forward prediction, and planning under constraints. The penalty kick is a world-model problem wearing athletic clothing.

This chapter tells the story of that idea. It is not a museum tour. Each of the four traditions contributed something that appears in many modern world models, and each exposed a failure mode that modern systems still face. Neuroscience and cognitive science supplied evidence and concepts for internal prediction and simulation. Cybernetics and control engineering supplied mathematics for state, feedback, observers, and optimal control over predicted futures. Classical AI planning supplied explicit models and search over their consequences; model-based reinforcement learning added the problem of learning models from interaction. The modern generative and foundation-model era supplied scale through predictors of rich sensory and latent futures.

We will move through these lineages roughly in historical order, but the point is not chronology. The point is that when you design a world model today, you are recombining decisions that these communities wrestled with for decades. Knowing where each idea came from tells you which assumptions it carries. Cognitive accounts assume that prediction error is a useful organizing signal. A latent dynamics model inherited from control theory assumes that a compact state exists and suffices. A generative video predictor inherited from computer vision assumes that matching sensory statistics is a useful training target. A Dyna-style planner inherited from reinforcement learning assumes that simulated experience can substitute for real experience. None of these assumptions is free, and the history below records where each has helped and where it has failed.

The term "world model" is used loosely across fields, and part of becoming fluent in this area is learning to hear which tradition a speaker is drawing from. A neuroscientist may mean a hippocampal map or cortical generative hierarchy; a control engineer, a plant model inside a controller; a reinforcement-learning researcher, learned transition dynamics; and a foundation-model researcher, a video generator or a language model's implicit physical knowledge. What Is a World Model? uses a narrower operational definition: a learned or constructed representation that predicts consequences of actions accurately enough to improve decisions. Some historical precursors below are broader predictive representations rather than complete action-conditioned world models under that definition. Their contribution is a component that later systems combine with action and decision machinery.

The table does not describe a monotonic march from narrow to general models. Each tradition chooses a modeled object for its own purpose. Neuroscience studies joint models of body and environment. Control theory models a particular plant, usually with equations derived or identified by engineers. Classical planners operate on task-specific symbolic operators, while model-based learning discovers task-relevant transition structure from experience. Foundation models learn from broad streams of sensory data before any one downstream task is fixed. The newest tradition therefore faces a particularly sharp version of an old question: what is the model for?

Two threads connect all four. First, prediction is often cheaper than action over the horizons that matter: evaluating a plan inside a model costs computation, while testing it in the world also costs time, energy, equipment, and sometimes survival. A slow or exceptionally detailed simulator can reverse that asymmetry, so it is a design constraint rather than a law. Where the asymmetry holds, it is the economic engine behind every tradition we visit. Evolution appears to have produced neural machinery that lets organisms learn without relying exclusively on dangerous trial and error. Engineers use simulators rather than letting a chemical plant fail while a controller learns. Reinforcement-learning agents use learned models when millions of real robot-hours would be impractical. Second, the model is never the world. Every tradition discovered, in its own vocabulary, that the gap between model and reality is where the interesting engineering lives. For Helmholtz, the gap was perceptual ambiguity resolved by priors. For Wiener, it was noise and delay that feedback had to absorb. In the Dyna lineage, it became model bias: planning inherits whatever errors the learned model contains, and simulated experience can propagate those errors into the policy, a compounding failure mode later model-based RL work made explicit. For today's video-generation debate, it is the difference between plausible footage and actionable physics. Keep both threads in mind as we proceed; nearly every development below is either a new way to buy cheap prediction or a new way to manage the model-world gap.

Long before anyone trained a neural network to predict the next frame, biologists and psychologists were cataloguing evidence that brains run predictive models of the world and of the body's own actions. This evidence accumulated over more than a century, across perception, motor control, spatial navigation, memory, and reasoning, and it converged on a conclusion that was far from obvious: the brain is not merely a reactive device that maps stimuli to responses, but also a predictive device that generates expectations and gives prediction errors a central role. This section covers the major strands of that evidence, because each strand maps onto a design decision you will face when building artificial world models. The mapping is worth making explicit up front. Unconscious inference foreshadows the encoder-decoder structure of latent-variable models. Efference copies motivate forward models used for state estimation and action-conditioned prediction. Cognitive maps anticipate state abstractions and successor representations. Mental simulation resembles imagined rollouts and planning in latent space. Predictive processing motivates hierarchical generative models trained by minimizing prediction error. The subsections trace these correspondences; later parts of the book develop the engineering versions in detail.

Hermann von Helmholtz developed the idea of unconscious inference across his Handbook of Physiological Optics, published in parts from 1856 to 1867. The retinal image is ambiguous: the same pattern of light can be compatible with many external scenes, and retinal motion can come from an object or from the observer's eye. Helmholtz argued that perception resolves such ambiguity by combining sensory evidence with learned expectations about its likely causes. "Unconscious" mattered: the inference was not deliberate reasoning, but a rapid process below awareness. The proposal was not Bayesian in modern notation, yet it supplied the historical structure that later Bayesian accounts of perception made explicit.

In modern notation, the brain estimates a latent cause $z$ from a sensation $x$ by weighting evidence against priors, exactly as Bayes' rule prescribes:

Read this equation slowly, because each factor corresponds to a recurring component of probabilistic world models. The likelihood $p(x \mid z)$ is the observation component: it describes how unobservable causes produce observable sensations. Together, the prior $p(z)$ and likelihood $p(x \mid z)$ define the joint generative model $p(x,z)=p(x \mid z)p(z)$. The posterior $p(z \mid x)$ is the perceiver's updated belief after seeing the data; the omitted normalizer $p(x)$ is the evidence that makes the posterior integrate to one. Perception, in this view, is not the beginning of cognition but the middle of it: it stands between a body of prior knowledge and the actions that knowledge supports.

Helmholtz wrote before Bayes' rule became a tool of psychology, and he used the word "inference" cautiously, but the structural claim is unmistakable: perception is a prediction problem run in reverse. The perceiver holds a model of how causes produce sensations, and perception is inference about causes given sensations. It took roughly 115 to 140 years for this idea to become computational through Bayesian models of perception in the 1980s and 1990s, depending on which of Helmholtz's writings marks the start and which early computational work counts as the arrival. The conceptual origin, however, lies in Helmholtz's nineteenth-century account.

This framing matters for world models because it separates two things that beginners often conflate: the generative direction (causes produce observations) and the inferential direction (observations are used to infer causes). Probabilistic latent-state architectures usually need both. The forward direction lets you simulate: given a hypothesized state of the world, roll out what you would see. The inverse direction lets you update your beliefs when reality disagrees with your predictions: given what you saw, revise your estimate of the state. An agent equipped with only the forward direction can imagine but cannot update its latent state from observations; an agent equipped with only the inverse direction can encode observations but cannot roll consequences forward. Kalman filters, variational autoencoders, and Dreamer's recurrent state-space model arrange these directions differently. Part II develops Bayesian filtering and belief states, while Part III develops latent-variable representations.

Move your eyes right now, from one word to the next. The visual scene sweeps across your retina, yet you do not experience the world as lurching. Now poke your eye gently through the eyelid: broadly similar retinal displacement is experienced as the world jumping. The inputs are not literally identical, because self-generated saccades also engage suppression and other preparatory mechanisms. The decisive distinction is that the first motion was commanded and predictable, while the second was imposed from outside. Information about the motor command helps the nervous system interpret the resulting sensory change.

In the 1950s, Erich von Holst and Horst Mittelstaedt formulated the reafference principle, and Roger Sperry independently described the same mechanism in fish, to explain this. When the brain issues a motor command, it also sends a copy of that command, an efference copy, to sensory areas. This copy lets the brain predict the sensory consequences of its own action and cancel out the predictable part, the reafference, leaving only the surprising part to be perceived. The elegance of the arrangement is that it solves two problems at once. It prevents the organism from being flooded by self-generated sensation, which would drown out signals from the external world. And it converts every self-produced movement into a test of the forward model: if the predicted sensory consequence matches the actual one, nothing needs to be reported; if it does not match, the mismatch is precisely the information the organism needs.

This is a forward model in the strictest sense: a mapping from current state and action to predicted next sensation. Work by Daniel Wolpert, Mitsuo Kawato, and colleagues in the 1990s made forward and inverse models a central computational account of human motor control; Wolpert and Kawato's 1998 paper is a primary statement of that framework. Before an object leaves a surface, people scale lifting forces to its expected weight; Johansson and Westling's unexpected-weight experiments showed the resulting error when that expectation was wrong. A deceptively light suitcase can therefore make the arm rise faster than intended. The cerebellum has been implicated in these predictions and in adapting movements when action and sensation no longer match, although that evidence does not make every cerebellar computation a forward model.

A charming demonstration of the same machinery: self-produced touch is usually much less ticklish than externally produced touch. When you try to tickle your own palm, your forward model predicts the sensation, and the brain attenuates it. Blakemore, Wolpert, and Frith's 1998 paper explicitly reports using fMRI and found reduced responses to self-produced touch; the bibliographic record gives Nature Neuroscience 1:635-640 and DOI 10.1038/2870. In a 1999 behavioral experiment, delaying the touch or perturbing its trajectory made it feel more ticklish as predicted and actual feedback diverged. The demonstration matters beyond its charm: the brain's response depends not just on the sensation but on how well it was predicted. Prediction quality modulates perception itself, a principle that returns in modern form as precision-weighting in predictive processing and uncertainty-aware filtering in engineering.

Notice what this buys the organism. A forward model converts a slow feedback loop (act, wait for sensation, correct) into a fast internal one (act, predict sensation, correct before the sensation even arrives). The external loop is limited by nerve conduction velocity, sensory transduction, and muscle activation, all of which are physically slow. The internal loop runs at the speed of neural computation, which is much faster. This is precisely the argument for model-based control that we will formalize in the cybernetics section, and it is the same argument behind model-predictive control and imagined rollouts in Part VII. When an engineer builds a Smith predictor or an MPC controller, she is installing an artificial efference copy: a prediction of what the plant will do in response to the command, available before the plant's sluggish dynamics deliver the real answer.

In the 1930s and 1940s, Edward Tolman ran experiments that behaviorism could not explain. The dominant psychology of the day held that learning was the formation of stimulus-response associations, stamped in by reward. On that view, an animal learns nothing without reinforcement, and what it learns is a habit, a fixed response to a fixed cue. Tolman's rats misbehaved in both respects. Rats allowed to wander a maze with no reward, and later rewarded at a goal location, navigated almost as well as rats trained with reward from the start. This latent learning meant the unrewarded rats had learned something about the maze's structure, not just a stimulus-response habit. They had been building knowledge the whole time, silently, waiting for a reason to use it. Other experiments showed detour behavior: rats familiar with a maze would take novel shortcuts when a barrier blocked the trained route, behavior that no chain of practiced stimulus-response pairs could produce. Tolman concluded in his 1948 paper "Cognitive maps in rats and men" that animals build internal representations of spatial layout, and that behavior is planned against these representations rather than emitted as fixed responses.

Tolman's proposal remained physiological speculation for more than two decades after his 1948 paper. Psychology had no tools for finding a map in tissue, and behaviorism remained hostile to internal states of any kind. Then in 1971, John O'Keefe and Jonathan Dostrovsky discovered place cells: neurons in the rat hippocampus with location-selective firing fields. Place-cell activity is not reducible to a single visual landmark or a particular sequence of turns; it relates to location, an abstract variable that multiple sensory streams and action sequences can imply, even as fields adjust with context and approach direction and remap entirely under sufficiently large contextual changes, as Muller and Kubie (1987) and Bostock, Muller, and Kubie (1991) demonstrated under controlled changes and experience. In 1978, O'Keefe and Lynn Nadel published The Hippocampus as a Cognitive Map, arguing that this circuit implements Tolman's construct. In 2005, more than three decades after the place-cell discovery, Edvard Moser, May-Britt Moser, and colleagues reported grid cells in the entorhinal cortex: neurons with strikingly regular hexagonal firing fields that appear to provide a metric coordinate system for space. O'Keefe and the Mosers shared the 2014 Nobel Prize in Physiology or Medicine for this work.

Two aspects of this research program deserve your attention as a world-model builder:

• The map encodes structure, not observations. A place cell can retain a spatial firing field across changes in illumination, approach direction, and some rearrangements, although sufficiently large contextual changes can cause remapping. The hippocampus therefore abstracts away from much raw sensory input toward state variables that support prediction and planning. This is an early biological analogue of what Chapter 2 calls a decision-sufficient representation: preserve variables that matter for decisions and discard others. A place field is not a photograph of the room. The later MuZero system made a related engineering choice by learning planning-relevant latent states without reconstructing every observation detail; it reappears in this chapter's history of learned models.
• The map supports inference beyond practiced routes. Tolman's detour and shortcut experiments showed that route memorization alone could not explain flexible navigation. Morris and colleagues' 1982 water-maze lesion experiments became an influential demonstration that hippocampal damage impairs flexible spatial navigation, although the strength of the deficit depends on the task and does not justify a blanket claim about every novel shortcut. Replay supplies a second clue. Wilson and McNaughton reported reactivation during sleep in 1994; Foster and Wilson recorded awake reverse replay in 2006; later work, including Gupta and colleagues in 2010, found sequences that could represent paths the animal had not just traversed. These results have motivated comparison with Dyna, the model-based reinforcement learning architecture introduced below: real experience updates a model, and internally generated sequences can support later decisions. The analogy is useful, but the recordings do not by themselves prove that replay performs a policy update.

The cognitive-map story also has a modern computational afterlife. Peter Dayan introduced the successor representation in 1993 in the context of temporal-difference learning. It is a matrix indexed by a current state and a possible successor state, with each entry recording the discounted expected occupancy of that successor under the agent's current policy: change the policy and the matrix changes with it. Kim Stachenfeld and colleagues showed in 2017 that successor representations predict features of hippocampal place-field geometry, including fields that skew opposite the direction of travel and distort around barriers as transition structure changes. In 2018, Timothy Behrens, Timothy Muller, James C. R. Whittington, and colleagues argued that cognitive maps may generalize beyond physical space to abstract relational structure. The engineering lesson is that a useful model can represent predictive relationships among states rather than reconstruct raw observations.

Cognitive science in the 1980s developed the idea that reasoning itself is simulation. Philip Johnson-Laird's Mental Models (1983) argued that people reason by constructing small internal models of a situation and inspecting them, rather than by applying logical rules to propositions. Asked whether "all the flowers are roses" follows from "some of the flowers are roses," people do not consult a proof system; they picture a garden with some roses and check whether other flowers could coexist with the statement. The mental model tradition explained systematic patterns in human reasoning errors as artifacts of what people's internal simulations leave out, which is a very different account from logic-based theories of thought. In cognitive psychology, the "theory theory" tradition associated with Alison Gopnik and Henry Wellman held that children learn causal models of the world and revise them like little scientists testing hypotheses. On this view, developmental learning is not accumulation of associations but theory change: the child's causal model of objects, minds, and living things undergoes revisions that resemble paradigm shifts, complete with periods of confusion when the old model fails and the new one is not yet stable.

In 2002, cognitive neuroscientist Georg Hesslow proposed that conscious thought can be understood as simulation of behavior and perception: when you deliberate, you covertly recruit some of the motor and perceptual machinery that would be engaged if you acted. Deliberation, on this account, is not symbol manipulation wholly detached from the sensorimotor system; it can be the sensorimotor system running offline, with overt action inhibited. Memory research supplied a complementary line of evidence. Schacter and Addis argued in 2007 that episodic memory is constructive and can recombine fragments of past experience into imagined futures. In the same year, Hassabis and colleagues reported that several patients with hippocampal amnesia had difficulty constructing coherent novel experiences, although later work showed that this deficit is not universal across all amnesic patients. Memory, in this view, is less like a recording archive than a parts bin for simulation. Endel Tulving had earlier called the capacity to mentally travel in subjective time "chronesthesia."

The simulation hypothesis gives a cognitive-science answer to the question "why do agents need world models?" Because deliberation can take the form of model-based rollouts. When you decide whether to say something in a meeting, you may simulate the conversation; when a chess player looks ahead, she runs a model of the game; when you route around traffic, you simulate the drive. The next two traditions pursued intelligent action through different mechanisms: cybernetics emphasized feedback, while AI planning emphasized search over predicted consequences. Together they show that explicit simulation is one tool among several, not the definition of intelligent action. It is worth being precise about what the cognitive evidence does and does not establish. It establishes that humans use simulation and that simulation can recruit machinery also used in perception and action. It does not establish that simulation is always beneficial: human simulation is also the source of anxiety, rumination, and systematic forecasting errors, and people often decide better when they rely on fast habitual responses than when they deliberate badly. The engineering translation is not "always simulate" but "simulate when the model is good enough and the stakes justify the compute," a conditional that recurs throughout this book.

The final cognitive strand assembles Helmholtz's inference, motor prediction, and attention into a single architectural proposal about the cortex. David Mumford suggested in the early 1990s that the cortex's hierarchical structure supports generative feedback: higher areas send predictions down, lower areas send prediction errors up. A motivating anatomical asymmetry was that feedback connections between cortical areas are abundant and, in many measured pathways, outnumber feedforward connections. Exact ratios vary by area, layer, and measurement method, but the prevalence of backward pathways is hard to reconcile with a purely feedforward account of perception. Rajesh Rao and Dana Ballard gave this idea a concrete computational form in 1999, showing that a hierarchical predictive-coding network reproduced response properties of visual neurons, including extra-classical receptive-field effects. Karl Friston, James Kilner, and Lee Harrison's 2006 account developed the idea into the free-energy principle, attempting to unify perception, action, learning, and attention as facets of prediction-error minimization, and Andy Clark's 2013 paper "Whatever next?" and 2016 book Surfing Uncertainty brought it to a wide audience.

Three claims in this framework matter for our purposes. First, action and perception share a loop: action is the control solution to the prediction problem, an idea that anticipates the control-as-inference view we will develop in Part II. Where classical accounts treat perception and action as separate pipelines, predictive processing treats them as two ways of resolving the same discrepancy between prediction and sensation: change the belief, or change the world. Second, one influential account holds that attention tracks uncertainty: precision-weighting amplifies prediction errors from reliable channels and discounts errors from noisy ones. This gives attention a possible computational role and anticipates the uncertainty-weighted fusion at the heart of the Kalman filter. Third, and most cautiously, the framework's grander ambitions, such as explaining all of cognition as free-energy minimization, remain contested. Critics have argued that the principle is too flexible to falsify, that prediction-error minimization cannot be the whole story of cortical function, and that the mathematical formulations admit multiple incompatible readings. The durable architectural idea is a hierarchy of generative predictions and corrective error signals. Residual connections, variational inference, and encoder-decoder loops are not direct implementations of predictive coding, but they can be read as engineering echoes of this predict-and-correct pattern.

While psychologists were inferring internal models from behavior, engineers were building external ones. This tradition contributed mathematics that the rest of the book runs on. The next chapter develops states, transition laws, trajectories, stability, and information-theoretic sufficiency; Part II later develops feedback and observers. Here we focus on how engineers came to put prediction inside controllers that operate under delay and uncertainty. Better sensors and faster actuators help in many regimes. They do not remove a substantial transport delay, however, and aggressive feedback across such a delay can destabilize the loop. In delay-dominated problems, prediction lets the controller act on where the system will be rather than only where it was when the measurement was produced. The goalkeeper from this chapter's opening is the biological version; the Smith predictor is the engineered archetype.

The story begins with an engineering embarrassment. In the 1860s, centrifugal governors that regulated steam engines (seemingly simple devices that throttle an engine when it speeds up) could oscillate badly on faster machines. Practitioners had empirical remedies, and Airy had analyzed the dynamics of a clockwork regulator for telescope drives in 1840, but there was no satisfactory general theory of feedback-loop behavior. James Clerk Maxwell's 1868 paper "On Governors" supplied the first systematic stability analysis of a broad class of governors. He showed that stability depends on the dynamics of the whole closed loop, not on the governor alone. A component that is well-behaved in isolation can destabilize a loop, while a loop containing an imperfect component can remain stable if its dynamics are arranged correctly. The lesson that the loop is the unit of analysis echoes through this book, from rollout divergence in learned models to receding-horizon control.

The conceptual leap came during World War II. Artillery had to hit moving aircraft, which meant predicting where the target would be when the shell arrived under noisy tracking. Pure reaction was too late, and naive extrapolation was fragile. This wartime prediction problem also drove Wiener's work on statistical prediction and filtering. Arturo Rosenblueth, Norbert Wiener, and Julian Bigelow addressed a related conceptual question in "Behavior, Purpose and Teleology" (1943): how to describe purposeful behavior without invoking a mysterious final cause. They argued that feedback gives purpose in animals and machines a mechanistic account. Wiener's 1948 book Cybernetics, subtitled "Control and Communication in the Animal and the Machine," named the field and argued that the same mathematics could illuminate thermostats, artillery trackers, and nervous systems. W. Ross Ashby's Design for a Brain (1952) treated adaptive behavior as regulation; his later An Introduction to Cybernetics (1956) stated the law of requisite variety, the more precise source for the claim that a regulator must match the relevant variety of its environment.

Cybernetics as a movement faded in the 1960s and 1970s, partly because its ambitions outran its tools. The mathematics of nonlinear, high-dimensional, uncertain systems did not yet exist, and the movement's grandest claims about society and cognition could not be cashed out. But its two central gifts survived intact. The first is the feedback loop as a unit of analysis: the insight that behavior arises from circular causality between a system and its environment, and that this circularity must be analyzed as a whole. The second is the recognition that regulation under delay and noise requires an internal representation of the regulated system. That recognition was about to be made precise.

Classical feedback control reacts to errors after they appear. Measure the output, compare it to the target, correct. That works when the loop is fast relative to the process. But many processes have long dead time: a shower where the water temperature responds seconds after you turn the knob, or a paper mill where a change at one end takes minutes to affect the other. Under dead time, aggressive or high-gain feedback can oscillate because the visible effect of the last action arrives late and the next correction targets an already changed situation. Low-gain feedback may remain stable but respond sluggishly. Anyone who has adjusted a shower temperature in waves of too hot and too cold has experienced the first regime.

Otto Smith's 1957 Smith predictor made the fix explicit: keep a model of the process inside the controller, simulate the effect of your control action on the model (which responds immediately, with no dead time), and base feedback on the model's immediate output, correcting it whenever a measurement arrives by comparing that measurement against the model's own delayed prediction. The controller regulates a prediction of the process rather than reacting to stale measurements directly. With an accurate process model, the dead time is moved outside the nominal feedback characteristic equation; model mismatch reintroduces delay sensitivity and limits robustness. This is an early, fully explicit instance of the pattern this book is about: a model inside the loop, serving as a fast surrogate for a slow world.

The 1960s generalized this from a trick to a theory. Rudolf Kálmán's 1960 filter solved a problem every world model faces: estimate an evolving state from noisy measurements. With linear dynamics, mutually independent Gaussian process and measurement noise, and a Gaussian initial state independent of both noise sequences, the Kalman recursion gives the minimum-mean-square-error estimate. If Gaussianity is replaced by finite second moments while those zero-mean, independence, and covariance assumptions are retained, the same recursion gives the best linear estimator. The filter maintains a mean and covariance and alternates between prediction through the model and correction from measurements. Prediction propagates belief and adds process uncertainty; correction contracts uncertainty according to measurement informativeness. David Luenberger's 1964 observer treated deterministic linear systems without requiring noise statistics. Its estimation error converges whenever the chosen observer gain makes the error dynamics stable; such a stabilizing gain exists precisely when the system is detectable. The obstruction is an unobservable unstable mode, not unobservability by itself, because unobservable modes that are already stable can decay without correction. In both methods, the internal model determines how prediction and observation are combined. This observer construction is distinct from the Francis-Wonham internal model principle of 1976, which concerns embedding disturbance or reference dynamics in a regulator.

The step-18 dashed line is intentionally difficult to separate from the solid trace. Once the disturbance ends, the measured state reanchors the rollout and the toy's internal dynamics again match the plant. That near-overlay is the recovery signal, not a missing third plan.

The next result is narrower than its memorable title suggests: it concerns a specific optimal-regulation setting and a correspondence between distinguishable regulator and system states.

In 1970, Roger Conant and W. Ross Ashby published a theorem with a provocative title: "Every good regulator of a system must be a model of that system." Their setting is specific: a deterministic system driven by stochastic disturbances, an objective that minimizes uncertainty in specified essential variables, and an optimal regulator considered in reduced form, without redundant distinguishable states. In that setting, the regulator's distinguishable states correspond homomorphically to distinguishable situations of the regulated system. A lookup table can meet this criterion when its indices distinguish the relevant situations and its stored outputs select actions; the model need not be a rich predictive simulator. Describing this as an estimator followed by a policy is a useful modern interpretation, not the theorem's literal statement.

The theorem should not be over-read. It does not say that the model must be explicit, learned, inspectable, or capable of multi-step simulation. Nor does it imply that every fixed-gain, memoryless controller contains a nontrivial model; only degenerate cases can satisfy a multi-state correspondence with a single undifferentiated controller state. As a design intuition, the theorem remains useful: in its setting, optimal regulation requires a correspondence between regulator states and the system distinctions relevant to the essential variables. That is narrower than the Francis-Wonham internal model principle and far narrower than a universal claim about brains or policies. The form and richness of the correspondence remain engineering choices.

The third tradition eventually made model acquisition an algorithmic problem. It began, however, with hand-written symbolic models: engineers specified states and operators, and programs searched over their consequences. Model-based reinforcement learning later replaced much of that specification with models learned from interaction. This changed validation as well as construction. A hand-written model can be inspected symbol by symbol; a learned model must be tested statistically, especially where experience is sparse. The lineage therefore runs from explicit symbolic planning to learned latent dynamics, and it connects directly to the model-based reinforcement learning chapters of Part II.

The first AI programs to "think ahead" did so over symbolic state spaces. Allen Newell, Clifford Shaw, and Herbert Simon demonstrated Logic Theorist in 1956 as a heuristic proof-search program. Their General Problem Solver, begun in 1957, introduced means-ends analysis: detect a difference between the current state and the goal, choose an operator that can reduce it, and recursively solve the preconditions for applying that operator. The procedure is recognizably a planning loop: evaluate the gap, imagine an action closing part of it, recurse. Here the world model is explicit: states, operators with preconditions and effects, and a search procedure that simulates operator application. Nothing is hidden; "imagination" is the application of an operator to a symbolic description to see what results.

Richard Fikes and Nils Nilsson's STRIPS (1971) gave this paradigm its durable formal shape: actions described by precondition, add, and delete lists, enabling planners that reason over action schemas rather than enumerating states. Under the usual blocks-world conventions, where variables bind to distinct movable blocks and the table is treated as a distinguished always-clear constant, a correct schema can write Move(X, Z, Y) with preconditions On(X,Z), Clear(X), and Clear(Y); add effects On(X,Y) and Clear(Z); and delete effects On(X,Z) and Clear(Y). Deleting On(X,Z) keeps one block recorded on at most one support; recording the old support Z as an argument allows the add effect to restore Clear(Z), freeing it for later moves. Moves that originate on the table require either admitting the table constant as Z under compatible predicates or adding a companion schema. This compression is the symbolic ancestor of a problem we will study in depth: how to represent dynamics compactly enough to plan over. The planning tradition matured into a deep field covering heuristic search, partial-order planning, and temporal and probabilistic planning. Under their stated assumptions, particular algorithms can guarantee completeness, return optimal plans, or make plans independently verifiable. Purely learned approaches do not generally provide the same guarantees, although hybrid systems can inherit them from a symbolic component. We will cover classical planning and optimal control in detail in Part II.

Two limitations of the symbolic tradition shaped what came next. First, the models were given, hand-written by engineers, which works for logistics and blocks worlds but not for a robot facing the physical world. Nobody can enumerate every precondition and effect of "grasp the cup" in a kitchen; the effects depend on friction, deformability, lighting, and contents. Second, crisp symbolic states can be brittle under noise: one wrong precondition may invalidate a plan. Classical planners did develop execution monitoring, plan repair, and replanning, so recovery was not absent, but these mechanisms still depended on a hand-specified symbolic interface that represented the relevant deviation. Learning models from data and making planning robust to model error became central problems.

Arthur Samuel's checkers program (1959) is remembered for self-play learning, but its more relevant innovation for us is the forward-looking backup: Samuel evaluated a position by simulating the game tree and propagating values backward. Richard Sutton later identified this as an early instance of temporal-difference learning, which he formalized in 1988: update a prediction of a future outcome toward the prediction that follows it. In finite tabular policy-evaluation problems where the behavior process visits every relevant state infinitely often and the step sizes satisfy the Robbins-Monro conditions, TD and Monte Carlo methods target the same policy value function, although their finite-sample updates differ. Constant step sizes typically bring the iterates to a bounded neighborhood of the target and leave them fluctuating within it, rather than producing almost-sure convergence to one value. With linear function approximation, Tsitsiklis and Van Roy (1997) characterize the on-policy TD fixed point as a projected solution. The combination of bootstrapping, function approximation, and off-policy data does not inherit that guarantee and can diverge; Baird (1995) gives classical linear counterexamples. Nonlinear approximation has additional failure modes, but they are not attributed to Baird's construction here. TD's practical advantage is that it uses intermediate transitions rather than waiting for a final return. It therefore underlies much of Part II and fits naturally wherever an agent learns chained predictions.

Sutton's Dyna architecture is the historical hinge between model-free and model-based reinforcement learning, and it is simple enough to state in one paragraph. An agent interacts with the real environment, storing each experienced transition in a learned model. Between real steps, it plans by sampling stored or imagined transitions and applying the same learning update it would apply to real experience. Real experience improves both the policy and the model; the model then turns each piece of real experience into additional simulated updates. The first conference account appeared in 1990, followed by the 1991 SIGART article linked here. The architecture dissolves what had seemed like a fundamental dichotomy. Model-free methods learn directly from experience; model-based methods learn a model and then plan. Dyna shows that interleaving direct and simulated updates can be effective; it does not claim that one fixed interleaving schedule is universally optimal.

Dyna's significance is not just architectural. It crystallized the trade-off that defines model-based reinforcement learning: real experience is expensive but trustworthy, simulated experience is cheap but only as good as the model. Moore and Atkeson's prioritized sweeping (1993) sharpened the point by propagating updates backward from transitions with large value changes, concentrating planning where updates can matter most. The biological echo is suggestive, although the chronology runs the other way: Wilson and McNaughton reported sleep reactivation in 1994, a few years after Dyna, and Foster and Wilson reported awake reverse replay in 2006. Later studies, including Singer and Frank (2009), found replay content related to reward and recent experience. These observations resemble prioritized simulated updates, but they do not establish that hippocampal replay implements Dyna or performs the same value update.

The Q-learning update at Dyna's core is worth writing down, since we will use it in the experiment below:

where $\alpha$ is a learning rate and $\gamma$ a discount factor. When $s'$ is terminal, the bootstrap term $\gamma \max_{a'} Q(s',a')$ is taken as zero. Read the update as a prediction-correction pair: the term $r + \gamma \max_{a'} Q(s', a')$ is a fresh estimate of the value of $(s,a)$ based on the observed successor for a real transition or the model-sampled successor for an imagined transition, and the bracket measures how wrong the old estimate was. The update nudges the estimate toward the fresher one by a fraction $\alpha$. In Dyna, this update runs both on real transitions $(s, a, r, s')$ and on transitions $(s, a, r, s')$ sampled from the learned model, and nothing in the update distinguishes the two cases.

For many years after Dyna, model-based reinforcement learning remained comparatively marginal in mainstream reinforcement learning, despite substantial work in planning, adaptive control, and probabilistic robotics. A persistent obstacle was model bias, which Part VI and Part XII treat in depth. If a learned model is wrong in a region the policy can visit, an optimizer may actively seek the error because it looks like an opportunity. A false shortcut or false prediction of safety can therefore produce confident failure. The 2010s brought methods that made this problem more manageable and model-based learning more competitive.

Marc Deisenroth and Carl Edward Rasmussen's PILCO (2011) attacked model bias with probabilistic models. Its Gaussian-process dynamics model propagated predictive uncertainty through imagined trajectories, and that uncertainty affected expected cost through moment matching rather than through a separate hand-added penalty. PILCO solved several benchmark control tasks from scratch with tens of seconds of interaction, a striking sample-efficiency result relative to contemporary model-free baselines. Guided policy search, introduced by Sergey Levine and Vladlen Koltun in 2013 and extended in work with Nolan Wagener, Pieter Abbeel, Chelsea Finn, and others, used model-based trajectory optimization to supervise reactive policies. The model could plan locally where it was valid, while the deployed policy acted without online model-based optimization.

The deep-learning era then collapsed the distinction between "model" and "perception." Two threads converged in the methods that followed.

• Learning in latent space. Rather than predicting every raw detail, learn a compressed latent state and model its dynamics. The World Models paper by David Ha and Jürgen Schmidhuber (2018) combined a variational-autoencoder-style vision model, a recurrent mixture-density network for latent dynamics, and a small controller trained inside the model's "dream." Those dream episodes were bounded, and the controller was evaluated in the real environment, because cheap simulation does not remove model bias or make arbitrarily long rollouts reliable.
• Planning with learned models at scale. MuZero (Julian Schrittwieser and colleagues, published in Nature in 2020) achieved superhuman play in Go, chess, and shogi, and strong Atari performance with a latent dynamics model that predicted rewards, values, policies, and planning-relevant transitions but did not reconstruct observations. It therefore showed that observation reconstruction was unnecessary for these tasks; stronger claims that reconstruction is generally harmful require a specified baseline and setting. AlphaZero, first released as a 2017 preprint and published in Science in 2018, planned over perfect game states. MuZero instead learned its own latent states and predicted only what planning consumed.

Meanwhile the Dreamer line (Danijar Hafner and colleagues: PlaNet 2019, Dreamer 2020, and DreamerV3, first released as a preprint in 2023 and published at ICLR 2024) made latent imagination practical for continuous control. The revised DreamerV3 paper reports one hyperparameter configuration across many domains and learning to collect diamonds in Minecraft without human demonstrations or a task-specific curriculum. This reduced dependence on per-task tuning; it did not eliminate every implementation or compute choice. Dreamer's recurrent state-space model, which Part V develops, combines an encoder-decoder, a stochastic forward model, and learning from imagined rollouts.

The newest tradition is the youngest, but its roots run through everything above. Its defining move is to treat the world model not only as an engineering component inside a control loop, but as a large generative model of experience itself, pretrained on broad data and later, or optionally, coupled to action. One branch emphasized passive predictive pretraining and deferred the acting question; another, visible early in robotics video prediction and the 2018 World Models architecture, coupled prediction to control from the outset. The section ends where those branches increasingly meet in world-action models. Whether broad predictive pretraining transfers reliably to decision-making remains open.

Deep generative modeling supplied the machinery. Variational autoencoders (Kingma and Welling, 2013) gave a tractable way to learn latent spaces with encoder-decoder structure. Generative adversarial networks (Goodfellow and colleagues, 2014) used a game-theoretic criterion that rewarded realism. Diffusion models in their modern formulation (Ho and colleagues, 2020), building on earlier diffusion and score-based work, learned to reverse a gradual noising process, and Ho and colleagues extended the approach to video in 2022. These methods learn data distributions, an important ingredient of a world model, but a passive generator is not yet action-conditioned. It can model what scenes tend to look like without identifying how an intervention changes the next state.

Ha and Schmidhuber's World Models paper of 2018 sits at the junction: its vision model (V) and memory model (M) are generative, but the architecture exists to serve a controller (C). Its signature demonstration, training the controller inside model-generated dreams, makes the decision coupling explicit. Robotics methods such as Finn and Levine's Deep Visual Foresight and Ebert and colleagues' self-supervised visual planning system had already pushed the same coupling further: predict future images conditioned on planned actions, then optimize a specified objective over the predicted video. Their controllers planned over short, finite prediction horizons rather than trusting arbitrarily long generated sequences. This is the defensible empirical lesson; a single frame count is not universal across architectures or experiments.

In 2024, OpenAI's Sora technical report framed large video generation models as "world simulators," arguing that scaling video prediction may be a path toward general-purpose world models. The ensuing debate has both conceptual and empirical parts: what counts as understanding physics, which conservation and causal failures current systems exhibit, and whether passive prediction or interactive data is needed for reliable action-conditioned models.

The optimistic reading is that predicting video well requires some implicit physical structure. Vast collections of internet video provide examples of motion, contact, deformation, and viewpoint change, although talking heads, screen recordings, animation, and edited footage carry very different amounts of usable signal about physical interaction. On this reading, scale may reduce the amount of hand-engineered structure required, but no defensible measurement establishes that all internet video is physically grounded or equally useful.

The skeptical reading is that video models trained on passive internet video optimize predictive plausibility rather than explicit counterfactual, action-conditioned accuracy. OpenAI's Sora technical report documents examples in which the model simulates physical interactions incorrectly or confuses spatial details over time. Those examples establish limitations of that system, not a failure rate for video models as a class. Passive video also contains few controlled counterfactual interventions. Whether scale closes that gap, or whether action conditioning and interactive data are essential, remains an open empirical question addressed in Part IX.

JEPA is an explicit modern statement of a lesson that long predates deep learning: the right prediction target need not be the raw observation, but can be a state abstraction that keeps what is predictable or needed for decisions and discards only what is neither. Predictive-state representations and formal state abstraction in reinforcement learning developed related ideas, building on earlier distinctions between useful and irrelevant state information. Pixel-level prediction forces the model to model leaves fluttering unpredictably, camera noise, and texture detail, spending capacity on entropy it cannot reduce; embedding-level prediction lets the model say "there will be a person walking there" without committing to the exact fold of their coat. The risk, symmetrically, is that a representation-space loss provides no gradient toward representing what the abstraction discarded, so the designer must ensure the abstraction keeps everything decisions will need. We will develop state abstraction in Part III and the JEPA family in Part IX.

The final development is the arrival of foundation models, large networks pretrained on broad data and adapted to many tasks. Language models can acquire substantial world knowledge: many answer questions about object properties, spatial relations, and everyday physics and support planning-style reasoning over described situations. A capable model will typically say that an ice cube on a hot stove melts, but reliability depends on the model, prompt, and requested precision. Whether that knowledge constitutes a world model in the operational sense is contested. Fluent verbal physics and usable quantitative physics are different achievements. Chapter 3's qualification test defines exactly four criteria: state sufficiency, action conditioning, counterfactual validity, and decision usefulness, which Chapter 3 names in those terms. The test therefore asks more than whether a model talks about the world plausibly.

The current frontier, and the subject of Part IX, is the convergence of these threads into world-action models: systems trained on interactive or video data to predict latent futures conditioned on actions, and to serve as the imagination engine for agents. Interactive video generators and neural game engines, including DeepMind's Genie and GameNGen, can be read as early, bounded instances: models whose outputs are not just videos but responsive environments. GameNGen, developed by researchers affiliated with Google Research and Tel Aviv University, runs a classic video game entirely inside a neural network that predicts frames conditioned on the player's inputs, so that playing the game means interacting with a learned model rather than executing programmed rules. Genie generates playable environments from unlabeled video by learning a latent action space that explains how scenes change. These systems blur the line between simulating a world and being one within the tasks, interaction windows, and data regimes on which they were demonstrated. They still face autoregressive drift, finite context, and limited action coverage. If an agent trains inside a learned environment, that environment's fidelity and coverage constrain what the agent can learn, alongside exploration, optimization, and policy capacity. If the twentieth century's question was "does the brain contain a model of the world?", and the late twentieth century's was "how should an agent learn one?", the current question is "how large can a learned world model get, and what does it fail at when it does?"

History makes claims; let's test one of them. Dyna argues that a learned model can extract more updates from each piece of real experience. We test that claim in a small NumPy gridworld by comparing Q-learning with Dyna under the same learning rule and hyperparameters. This is a Dyna experiment, not a miniature DreamerV3: Dyna interleaves replayed model transitions with direct updates, whereas DreamerV3 trains policies through imagined trajectories inside a learned latent model. The loose lineage is their shared use of learned dynamics to turn real experience into additional learning signal. Planning consumes extra random draws, so paired seeds do not produce identical trajectories; averaging twenty independent seeds reduces that sampling noise while leaving planning steps as the systematic treatment difference.

The environment is an obstacle-free $5 \times 5$ grid. The agent starts in the top-left corner and must reach the goal in the bottom-right. Each non-goal step costs $-1$ reward; the transition entering the goal receives $+20$ instead of the step cost and ends the episode. With probability $0.1$ the environment replaces the commanded action with one sampled uniformly from all four actions. The commanded action therefore still executes with probability $0.9 + 0.1/4 = 0.925$. The optimal no-slip episode return is $20 - 7 = 13$, since the shortest path takes eight actions and only the first seven receive $-1$. The executed display below is the canonical record of the twenty-seed outcomes.

First, the environment:

Read these two figures together as a small illustration of the computation-for-experience trade. Simulated transitions accelerate learning in this gridworld under the declared seeds, while the labeled sweep shows how the measured reduction in real interactions varies with planning depth. Differences of one reporting interval should not be read as a general law because the threshold is sampled only every 50 steps. Planning cannot add observations that the model has never received, but the sweep alone does not prove that coverage is the active bottleneck.

The key parameters for the Dyna experiment are:

• alpha = 0.3: Q-learning step size for both direct and planning updates.
• gamma = 0.95: Discount factor. The conventional horizon scale $1/(1-\gamma)=20$ is roughly two and a half times the grid's eight-action shortest path when every commanded move succeeds, and the goal signal retains a factor of $0.95^8 \approx 0.66$ along that slip-free path.
• eps = 0.05: Exploration probability of the online epsilon-greedy behavior policy that also generates the plotted returns.
• planning_steps = 0 or 10: Simulated updates between real steps in the main comparison; the sweep uses 0, 1, 5, 10, 50.
• n_steps = 3000: Real environment interactions per run.
• timeout = 60: Maximum episode length before a time-limit reset. Timeouts retain a bootstrap value; only the goal is terminal.
• SLIP = 0.1: Probability of replacing the command with an action drawn uniformly from all four actions, leaving the commanded action's total execution probability at 0.925.
• GRID = 5, START = (0, 0), GOAL = (4, 4): Environment geometry behind the eight-action shortest path.
• THRESHOLD = 7.0, STEP_GRID = 100, 150, ..., 2950: Reporting threshold and its 50-step measurement resolution.
• N_SEEDS = 20: Independent seeds averaged into each curve. A formal comparison would also report confidence intervals or other uncertainty estimates.

The four assumptions named in the opening have now met their historical tests. Prediction error organized Helmholtzian and predictive-processing accounts, but only under an inference model. Compact state powered cognitive maps and control, but only when the abstraction retained what the task needed. Matching sensory statistics enabled modern generators, but did not by itself supply interventions or decisions. Simulated experience accelerated Dyna in the experiment above, but only over transitions already represented by the learned model. The rest of the book develops these conditions rather than treating any of the four assumptions as universally valid.

It is worth pausing on what this history does not show, because the failures are as instructive as the successes. Three related failure modes recur in different forms across the neuroscience, control, reinforcement-learning, and foundation-model discussions above. The parallels are conceptual rather than evidence that every field encountered an identical empirical failure. Recognizing the family resemblance is useful preparation for recognizing these problems in your own designs.

The first recurring failure is the gap between predicting and controlling. Helmholtz's perceiver, the Kalman filter, and Sora all predict; none of them, by itself, decides. Several traditions stalled when prediction quality was treated as sufficient for control. A video model can produce visually plausible sequences yet still lack the action interface and intervention data needed to answer controlled counterfactuals; this is a design risk, not a measured verdict about every video generator. The Smith predictor's dead-time compensation and MPC's receding horizon were earlier responses to the same underlying issue. The pattern is subtle because prediction quality does help control: a better predictive model makes every downstream decision better informed. But the help is bounded, and past the bound, additional predictive fidelity buys nothing that decisions can use. A video model that renders hair motion flawlessly may have spent capacity a particular planner cannot use. The discipline that resolves it, separating open-loop prediction quality from closed-loop decision usefulness, is exactly the discipline Chapter 3's four-criterion qualification test enforces.

The second recurring failure is model exploitation and compounding error. Dyna's planning amplifies whatever the model believes, right or wrong. MuZero reduces one source of burden by learning a representation for planning without reconstructing observations, while PILCO propagates predictive uncertainty through expected cost; neither removes model error. One-step errors accumulate differently depending on their structure. In an illustrative additive calculation, a constant signed bias of 0.01 per step sums to $0.01t$, while independent zero-mean errors with standard deviation 0.01 have root-mean-square sum $0.01\sqrt{t}$. State-dependent dynamics can contract or amplify either pattern. Uniformly sampled evaluation understates planner-facing error when the planner concentrates its queries in sparse, higher-error regions. That is the methodological point here; the relevant empirical check is error under the planner-induced state-action distribution.

The third lesson is about representation. Place cells, successor representations, MuZero's latents, and JEPA's embedding-space predictions all converge on the same conclusion: the useful model is not of observations but of a state abstraction that preserves what is predictable and decision-relevant. This is not an implementation detail; it is arguably the central design problem, and Part III is devoted to it. Notice that the convergence spans mechanisms as different as biological evolution, dynamic programming, and gradient descent, which is some evidence that the conclusion reflects the structure of the problem rather than the habits of any one community. What to keep, what to discard, and what granularity to predict at are choices that determine everything downstream: how much data the model needs, how far it can roll out, and whether its outputs can support decisions at all.

The impact side of the ledger is equally clear. Modern systems combine ideas from these traditions: action-conditioned prediction from motor neuroscience, state estimation and receding-horizon planning from control, learning through imagined experience from model-based RL, and generative pretraining at scale. Dreamer is one such composition: an encoder and recurrent belief state feed a stochastic latent dynamics model, and actor-critic learning uses imagined latent trajectories. That architecture echoes Helmholtzian inference, observer-style filtering, and Dyna's use of learned dynamics, but it is not literally any one of them. The assumptions can also conflict, as when a reconstruction objective favors perceptual detail while a planning objective favors decision relevance. The next chapter develops the mathematical foundations needed to reason about those choices: conditional distributions, states and transition laws, trajectories, stability and attractors, and information-theoretic sufficiency.

The dates and attributions above follow the primary and foundational records below. The list is grouped by lineage and gives a resolvable link for dated papers wherever one is available; books are identified by author, year, and title. It is a verification trail for the chapter's central historical claims, not a claim that every interpretive sentence has a single source.

• Perception, motor prediction, and spatial maps: Helmholtz, Handbuch der physiologischen Optik, original fascicles published by Leopold Voss in 1856, 1860, and 1867; von Holst and Mittelstaedt (1950), Das Reafferenzprinzip; Sperry (1950), “Neural basis of the spontaneous optokinetic response”; Johansson and Westling (1988), expected-weight programming of grip forces; Tolman and Honzik (1930), “Introduction and removal of reward, and maze performance in rats,” University of California Publications in Psychology 4:257-275; Tolman, Ritchie, and Kalish (1946), orientation and shortcut learning; Tolman (1948), “Cognitive maps in rats and men”; O'Keefe and Dostrovsky (1971), place units; O'Keefe and Nadel (1978), The Hippocampus as a Cognitive Map; Muller and Kubie (1987), environmental control of place-cell firing; Bostock, Muller, and Kubie (1991), experience-dependent place-cell remapping; Hafting et al. (2005), grid cells; the 2014 Nobel Prize in Physiology or Medicine; Morris et al. (1982), hippocampal lesions in the water maze; Wilson and McNaughton (1994), sleep reactivation; Foster and Wilson (2006), awake reverse replay; and Gupta et al. (2010), replayed paths not reducible to recent experience, Neuron 65(5):695–705.

• Predictive representations and mental simulation: Dayan (1993), the successor representation; Stachenfeld, Botvinick, and Gershman (2017), a predictive map; Behrens et al. (2018), abstract cognitive maps; Wolpert, Ghahramani, and Jordan (1995), an internal model for sensorimotor integration; Wolpert and Kawato (1998), paired forward and inverse models; Blakemore, Wolpert, and Frith (1998), “Central cancellation of self-produced tickle sensation,” Nature Neuroscience 1:635-640; Blakemore, Frith, and Wolpert (1999), spatiotemporal perturbations of self-produced touch; Johnson-Laird (1983), Mental Models; Hesslow (2002), conscious thought as simulation; Schacter and Addis (2007), constructive episodic simulation; Hassabis et al. (2007), hippocampal amnesia and imagined experience; and Tulving (2002), “Episodic memory: From mind to brain”.

• Predictive processing: Mumford (1992), corticocortical loops; Rao and Ballard (1999), predictive coding in visual cortex; Friston, Kilner, and Harrison (2006), a free-energy account; and Clark (2013), predictive brains. A later development relative to Helmholtz's nineteenth-century tradition is the computational-Bayesian framing represented by Knill and Richards (1996), Perception as Bayesian Inference.

• Control, observers, and predictive control: Airy (1840), “On the Regulator of the Clock-work for effecting uniform Movement of Equatoreals,” Monthly Notices of the Royal Astronomical Society 5(5):41-43; Maxwell (1868), governors; Rosenblueth, Wiener, and Bigelow (1943), behavior and purpose; Wiener (1948), Cybernetics; Ashby (1952), Design for a Brain, and Ashby (1956), An Introduction to Cybernetics; Smith (1957), “Closer control of loops with dead time,” Chemical Engineering Progress 53(5):217-219; Kalman (1960), linear filtering; Luenberger (1964), observers; Francis and Wonham (1976), the internal-model principle; Conant and Ashby (1970), good regulators; Richalet et al. (1978), model predictive heuristic control; Cutler and Ramaker (1980), “Dynamic matrix control: A computer control algorithm,” Proceedings of the Joint Automatic Control Conference, WP5-B; Mayne et al. (2000), constrained MPC stability and robustness in Automatica; and Qin and Badgwell (2003), industrial MPC.

• Symbolic planning and model-based reinforcement learning: Newell, Shaw, and Simon (1956), the Logic Theory Machine, followed by their 1959 report on the General Problem Solver; Fikes and Nilsson (1971), STRIPS; Samuel (1959), machine learning in checkers; Sutton (1988), temporal-difference learning; Sutton (1990), “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming,” ICML, pp. 216-224, and the 1991 SIGART article; Watkins (1989), Learning from Delayed Rewards, and Watkins and Dayan (1992), Q-learning; Moore and Atkeson (1993), prioritized sweeping; Baird (1995), linear function-approximation instability; Tsitsiklis and Van Roy (1997), linear TD convergence; Deisenroth and Rasmussen (2011), PILCO; Levine and Koltun (2013), guided policy search; Ha and Schmidhuber (2018), World Models; Silver et al. (2018), AlphaZero; Schrittwieser et al. (2020), MuZero; PlaNet (2019); Dreamer (2020); and DreamerV3, revised from its 2023 preprint and published at ICLR 2024.

• Generative and foundation-scale models: Kingma and Welling (2013), variational autoencoders; Goodfellow et al. (2014), GANs; Ho et al. (2020), denoising diffusion; Ho et al. (2022), video diffusion; Finn and Levine (2017), action-conditioned visual foresight; Ebert et al. (2017), self-supervised visual planning; LeCun (2022), JEPA and autonomous intelligence; V-JEPA; OpenAI's Sora technical report; Genie; and GameNGen.

• The four lineages contributed different working tools and exposed different limits. Neuroscience supplied prediction, simulation, and state-like representations but also showed context dependence and ambiguity. Control supplied feedback, estimation, and receding horizons but made model mismatch explicit. Symbolic planning and model-based RL supplied search and imagined updates but exposed hand-authored-model brittleness and learned-model bias. Generative and foundation models supplied scale but sharpened the distinction between plausible prediction and decision-useful, action-conditioned dynamics.
• The idea of internal models is old and convergent. Helmholtz's unconscious inference, von Holst and Mittelstaedt's efference copies, Tolman's cognitive maps, and predictive processing all use internal models to explain prediction under partial observation. Tolman's and later predictive-processing traditions extend this core toward simulation and planning, while efference-copy accounts emphasize predictions tied to the self's actions. Their convergence across distinct but partly connected literatures suggests that internal prediction addresses a recurring problem of acting with incomplete observations.
• Control theory made the intuition exact under stated assumptions. The Smith predictor, Kalman filter, Luenberger observer, and MPC showed how a model can support regulation under delay and noise, while Conant and Ashby formalized an internal state correspondence for optimal regulation in their setting. Receding-horizon replanning is one battle-tested way to limit open-loop trust in an imperfect model; robust MPC and other uncertainty-aware methods add stronger guarantees.
• Classical AI planning supplied explicit models and search; model-based reinforcement learning added learning from interaction. Samuel's backups, Dyna's replayed model transitions, MuZero's planning representation, and Dreamer's latent imagination are related but distinct ways to convert predictions into better decisions. Their benefit is bounded by the accuracy and support of the learned model.
• The foundation-model era changed the scale, not the questions. Generative video models, JEPA-style latent predictors, and world-action models are new instantiations of the same design axes: what to predict, in which space, conditioned on which actions, and evaluated for which use. Scale shifts where the trade-offs bite; it does not remove them.
• The Dyna experiment makes the economics concrete. In the deterministic twenty-seed execution, Dyna reaches the displayed return threshold with fewer real interactions than Q-learning. The labeled planning-depth sweep records how that interaction cost changes; because crossings are measured on a 50-step grid, one-interval differences should not be overinterpreted.
• The recurring failure modes concern representation, knowledge, and sequential dynamics: confusing prediction quality with decision usefulness, exploiting model errors, and compounding rollout error. Many design decisions in the rest of this book respond to one or more of these problems.

For our purposes, the historical case is established: models help agents predict, infer, and plan, but only under conditions that the next chapters will make explicit. The next chapter builds the machinery for conditional prediction, state transitions, trajectories, stability, and sufficient representations.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the intellectual and cognitive history of world models.