Intellectual and Cognitive History › Generative Models and the Foundation-Model Era

The newest tradition is the youngest, but its roots run through everything above. Its defining move is to treat the world model not only as an engineering component inside a control loop, but as a large generative model of experience itself, pretrained on broad data and later, or optionally, coupled to action. One branch emphasized passive predictive pretraining and deferred the acting question; another, visible early in robotics video prediction and the 2018 World Models architecture, coupled prediction to control from the outset. The section ends where those branches increasingly meet in world-action models. Whether broad predictive pretraining transfers reliably to decision-making remains open.

Deep generative modeling supplied the machinery. Variational autoencoders (Kingma and Welling, 2013) gave a tractable way to learn latent spaces with encoder-decoder structure. Generative adversarial networks (Goodfellow and colleagues, 2014) used a game-theoretic criterion that rewarded realism. Diffusion models in their modern formulation (Ho and colleagues, 2020), building on earlier diffusion and score-based work, learned to reverse a gradual noising process, and Ho and colleagues extended the approach to video in 2022. These methods learn data distributions, an important ingredient of a world model, but a passive generator is not yet action-conditioned. It can model what scenes tend to look like without identifying how an intervention changes the next state.

Ha and Schmidhuber's World Models paper of 2018 sits at the junction: its vision model (V) and memory model (M) are generative, but the architecture exists to serve a controller (C). Its signature demonstration, training the controller inside model-generated dreams, makes the decision coupling explicit. Robotics methods such as Finn and Levine's Deep Visual Foresight and Ebert and colleagues' self-supervised visual planning system had already pushed the same coupling further: predict future images conditioned on planned actions, then optimize a specified objective over the predicted video. Their controllers planned over short, finite prediction horizons rather than trusting arbitrarily long generated sequences. This is the defensible empirical lesson; a single frame count is not universal across architectures or experiments.

In 2024, OpenAI's Sora technical report framed large video generation models as "world simulators," arguing that scaling video prediction may be a path toward general-purpose world models. The ensuing debate has both conceptual and empirical parts: what counts as understanding physics, which conservation and causal failures current systems exhibit, and whether passive prediction or interactive data is needed for reliable action-conditioned models.

The optimistic reading is that predicting video well requires some implicit physical structure. Vast collections of internet video provide examples of motion, contact, deformation, and viewpoint change, although talking heads, screen recordings, animation, and edited footage carry very different amounts of usable signal about physical interaction. On this reading, scale may reduce the amount of hand-engineered structure required, but no defensible measurement establishes that all internet video is physically grounded or equally useful.

The skeptical reading is that video models trained on passive internet video optimize predictive plausibility rather than explicit counterfactual, action-conditioned accuracy. OpenAI's Sora technical report documents examples in which the model simulates physical interactions incorrectly or confuses spatial details over time. Those examples establish limitations of that system, not a failure rate for video models as a class. Passive video also contains few controlled counterfactual interventions. Whether scale closes that gap, or whether action conditioning and interactive data are essential, remains an open empirical question addressed in Part IX.

JEPA is an explicit modern statement of a lesson that long predates deep learning: the right prediction target need not be the raw observation, but can be a state abstraction that keeps what is predictable or needed for decisions and discards only what is neither. Predictive-state representations and formal state abstraction in reinforcement learning developed related ideas, building on earlier distinctions between useful and irrelevant state information. Pixel-level prediction forces the model to model leaves fluttering unpredictably, camera noise, and texture detail, spending capacity on entropy it cannot reduce; embedding-level prediction lets the model say "there will be a person walking there" without committing to the exact fold of their coat. The risk, symmetrically, is that a representation-space loss provides no gradient toward representing what the abstraction discarded, so the designer must ensure the abstraction keeps everything decisions will need. We will develop state abstraction in Part III and the JEPA family in Part IX.

The final development is the arrival of foundation models, large networks pretrained on broad data and adapted to many tasks. Language models can acquire substantial world knowledge: many answer questions about object properties, spatial relations, and everyday physics and support planning-style reasoning over described situations. A capable model will typically say that an ice cube on a hot stove melts, but reliability depends on the model, prompt, and requested precision. Whether that knowledge constitutes a world model in the operational sense is contested. Fluent verbal physics and usable quantitative physics are different achievements. Chapter 3's qualification test defines exactly four criteria: state sufficiency, action conditioning, counterfactual validity, and decision usefulness, which Chapter 3 names in those terms. The test therefore asks more than whether a model talks about the world plausibly.

The current frontier, and the subject of Part IX, is the convergence of these threads into world-action models: systems trained on interactive or video data to predict latent futures conditioned on actions, and to serve as the imagination engine for agents. Interactive video generators and neural game engines, including DeepMind's Genie and GameNGen, can be read as early, bounded instances: models whose outputs are not just videos but responsive environments. GameNGen, developed by researchers affiliated with Google Research and Tel Aviv University, runs a classic video game entirely inside a neural network that predicts frames conditioned on the player's inputs, so that playing the game means interacting with a learned model rather than executing programmed rules. Genie generates playable environments from unlabeled video by learning a latent action space that explains how scenes change. These systems blur the line between simulating a world and being one within the tasks, interaction windows, and data regimes on which they were demonstrated. They still face autoregressive drift, finite context, and limited action coverage. If an agent trains inside a learned environment, that environment's fidelity and coverage constrain what the agent can learn, alongside exploration, optimization, and policy capacity. If the twentieth century's question was "does the brain contain a model of the world?", and the late twentieth century's was "how should an agent learn one?", the current question is "how large can a learned world model get, and what does it fail at when it does?"