Mathematical and Dynamical Foundations

In Chapter 2 we described a world model informally: an encoder that turns observations into a state, a dynamics model that predicts how that state evolves, and a way of reading rewards or outcomes back out. Every one of those pieces rests on a small body of mathematics that this chapter puts in place. You do not need years of control theory to follow the rest of this handbook, but you do need four ideas to be solid: how we represent uncertainty with random variables and conditional distributions, how states evolve through transitions to form trajectories, how those trajectories settle onto fixed points and attractors, and how information theory tells us which parts of the past a state actually needs to carry.

It is worth pausing on why precisely these four. The first idea supplies the language: without conditional distributions you cannot even state what a world model claims about the future. The second supplies the object: trajectories built from local transition laws are what a world model produces and what training data consist of. The third supplies the diagnostic: long-horizon behavior is governed by stability and attractors, and ignoring them is the single most common way a technically competent model fails in deployment. The fourth supplies the design principle: a finite state cannot hold an infinite past, so every practical world model is an exercise in principled forgetting, and information theory is what turns "forgetting" from a euphemism into a measurable quantity.

We will lean on one running example throughout: a thermostat controlling the temperature of a room. It is almost embarrassingly simple, yet it contains a stochastic transition law, a limit-cycle attractor, a hidden state variable that temperature alone cannot capture, and a sharp lesson about the difference between predicting well one step ahead and reproducing the behavior of a system over long horizons. By the end of the chapter you will have simulated it, fit models to it, and watched a model with excellent one-step accuracy fail in closed loop for reasons that are purely dynamical. The thermostat earns its place precisely because nothing about it is exotic: if these failure modes appear in a system with one hidden bit, they will appear with interest in robotics, finance, and language.

If you have seen probability and linear algebra before, treat this chapter as a calibration pass: the goal is not novelty but shared vocabulary, stated assumptions, and a few intuitions, especially about stability and compounding error, that most machine-learning curricula skip entirely. Readers encountering the material for the first time should work through the code cells rather than skimming them; each one is short, and each one demonstrates a claim that would otherwise remain an assertion.

A world model is fundamentally a device for answering questions of the form "what might happen next, and with what probabilities?" Before we can talk about transitions and trajectories, we need a precise language for that kind of question. That language is probability theory, and its central objects are random variables and the conditional distributions that connect them. Everything else in this chapter, and most of what follows in this handbook, is built by composing these objects in time.

A random variable is not a variable in the programming sense. It is a formal device for attaching numbers to the outcomes of something uncertain, together with a distribution that says which numbers are more or less likely. The distinction matters because programmers routinely conflate the two: a Python variable holds one value at a time, while a random variable holds a whole space of possible values at once, weighted by belief. When we say a room's temperature tomorrow is a random variable, we are not saying the temperature is fuzzy or ill-defined; we are saying that our knowledge of it is incomplete, and the distribution is a complete accounting of that incompleteness.

When we write $s_{t+1} \sim p(\cdot \mid s_t, a_t)$ later in this chapter, the tilde means "drawn from," and the whole expression says: the next state is a random variable whose distribution depends on the current state and action. Nothing more mysterious than that. The dot inside the distribution is a placeholder for the value being drawn, and the part after the bar lists everything the draw is conditioned on. Learning to parse such expressions fluently pays off immediately, because nearly every formula in this handbook is a statement about who is conditioned on whom.

Two habits worth building now. First, always ask what a distribution is over: temperatures, pixel intensities, latent vectors, actions. Distributions over different spaces behave differently, support different operations, and demand different estimators, and many modeling bugs amount to silently mixing up which space a distribution lives in. Second, always ask what a distribution is conditioned on: a predictive distribution conditioned on the wrong information is worse than useless, because it feels principled while answering the wrong question. A weather model conditioned only on the season will produce confident seasonal averages; conditioned on today's pressure field, it produces something you would actually act on. The difference between those two conditionings is, in miniature, the difference between a bad world model and a good one.

Suppose $X$ and $Y$ are two random quantities, for example today's soil moisture and tomorrow's plant growth. Their joint distribution $p(x, y)$ describes them together: it assigns a probability, or probability density, to every combination of values, and thereby encodes not just how each quantity varies on its own but how they co-vary. Summing or integrating one out gives the marginal:

The marginal answers questions about $Y$ alone, deliberately ignoring whatever $X$ might have told us. It is the distribution you are stuck with when you have no access to $X$, and its spread is typically wider than what conditioning would leave you.

The conditional distribution reverses the direction of reasoning. It answers: if I have already observed $X = x$, what should I now believe about $Y$?

defined whenever $p(x) > 0$. Read as a rule, it says: restrict attention to the slice of the joint where $X$ takes the observed value, then renormalize so the slice sums to one. Conditioning is the mathematical core of every world model. An encoder maps an observation to a state; a dynamics model produces $p(s_{t+1} \mid s_t, a_t)$; a reward head produces $p(r \mid s_t, a_t)$. All of these are conditional distributions, and the quality of a world model is largely the quality of its conditionals. When this handbook later speaks of calibration, uncertainty quantification, or model exploitation by planners, it is always asking the same underlying question: are these conditionals honest?

Conditioning changes shapes, not just centers. If $Y = 0.8X + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0, 0.5^2)$, then knowing $X = 1$ does not merely shift your belief about $Y$ toward $0.8$; it also shrinks its spread from the marginal spread of $Y$ down to $0.5$. Both effects follow from the same algebra: once $X$ is known, the only remaining randomness is the noise term, so the conditional distribution is the noise distribution relocated to the regression line. Let's see both effects empirically.

The reason conditional distributions deserve this much attention is that they compose. Any joint distribution over a sequence can be factored into a product of conditionals, a fact called the chain rule of probability:

This identity looks innocuous and is anything but. It says that a full distribution over an entire trajectory can be built from nothing but a sequence of local predictive distributions, each conditioned on the past. There is no additional assumption hiding in it: the chain rule is an identity, true for every joint distribution whatsoever. All of the modeling content enters through the choices we make about what each conditional may depend on, which is exactly where assumptions like Markovianity will enter shortly. Every autoregressive language model, every recurrent dynamics model, and every rollout-based planner in this handbook is an application of the chain rule. When a world model rolls out a prediction for a hundred steps, it is multiplying a hundred conditional distributions together, and the failure modes of that multiplication, which we will quantify shortly, are among the central practical problems of the field.

Expectation is the other workhorse. For a distribution $p(x)$, the expected value is $\mathbb{E}[X] = \sum_x x\,p(x)$ or $\int x\,p(x)\,dx$: the probability-weighted average of the random variable, the single number that summarizes where the distribution's mass sits. Two properties carry most of the weight in practice. The law of iterated expectations, $\mathbb{E}[Y] = \mathbb{E}\big[\mathbb{E}[Y \mid X]\big]$, says that averaging conditional predictions reproduces the marginal prediction, which is why a simulator's average behavior can be correct even when no individual rollout is. This is simultaneously reassuring and dangerous: reassuring because aggregate statistics computed from simulations can be trusted even when individual runs look strange, and dangerous because it means average-case validation can hide pathologies that only appear trajectory by trajectory. And linearity of expectation, $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$, holds even when $X$ and $Y$ are dependent, which is why error budgets can often be added component-wise even when error sources interact. That linearity holds without independence is what keeps many applied probability calculations tractable.

With the probabilistic vocabulary in place, we can now define the object a world model maintains: a state that evolves through time according to a transition law, generating trajectories. The definitions in this section look elementary, but each one embeds a modeling decision that practitioners get wrong constantly, so we will be explicit about what each term commits us to.

Throughout this handbook we keep four concepts strictly separate, and it pays to fix them now:

• Observation: what the agent's sensors deliver at time $t$, written $o_t$. Observations may be incomplete, delayed, noisy, or redundant. A camera sees surfaces but not velocities; a thermometer reports temperature but not the heater's switch position.
• State: the information used to summarize the past for purposes of predicting the future, written $s_t$. The state may be observed directly, constructed by an encoder, or inferred as a belief. The state is defined by its predictive job, not by its physical interpretation.
• Action: the input the agent chooses, written $a_t$. Actions are the handle by which the agent influences the transition law, and their presence is what separates prediction from control.
• Environment: the external process that consumes $(s_t, a_t)$ or $(o_t, a_t)$ and produces the next observation, along with any reward. The environment is the ground truth against which every model in this handbook is ultimately judged.

A state is good not because it resembles the world internally but because it is useful for prediction. This pragmatic attitude, which goes back to the cybernetic tradition discussed in Chapter 5, lets us sidestep philosophical questions about what the "real" state of the world is. A thermostat's state can be two numbers and a switch position; it does not need to contain the position of every air molecule in the room. The molecular detail is real, but it is irrelevant to the predictive question the controller faces, and carrying it would be pure overhead. This criterion, usefulness for prediction, is the thread that connects the state concept to the sufficient statistics of the final section: a good state is precisely a summary that keeps what matters and discards what does not.

A transition law specifies how the state evolves. In the simplest case it is a deterministic function:

Deterministic transitions are an idealization. Real systems are buffeted by unmodeled influences: sensor noise, discretization error, and genuine randomness. The general treatment is a transition kernel, a conditional distribution over next states:

The deterministic case is the special case where this distribution puts all its mass on one point. A useful mental model is that $f(s_t, a_t)$ predicts the center of the next-state distribution while the kernel also carries its spread. Models that track only the center, which is what most neural network dynamics models do by default, silently assume the spread does not matter. Over long horizons, it always does: spread is what tells a planner whether a risky maneuver has a five percent or a fifty percent chance of failure, and a model blind to spread answers that question with confident nonsense.

The Markov property deserves emphasis because it is the hinge between this chapter and nearly everything that follows. It is what allows a world model to be a recurrence rather than a function of ever-growing histories, and it is what makes dynamic programming, filtering, and reinforcement learning tractable at all. Without it, every prediction problem would require re-processing the entire history from scratch, and the computational cost of planning would grow without bound as experience accumulates. With it, the entire past is compressed into a fixed-size object, and all reasoning proceeds forward from there. When we reach Part II: Inference, Decisions, and Control, the Markov decision process will formalize exactly this structure with actions and rewards attached. When the state you have is not Markov, the fixes are either to enrich the state, which is the subject of state abstraction in Part III, or to carry an explicit belief over hidden states, which is the subject of Bayesian filtering. The thermostat later in this chapter gives a concrete taste of the first remedy: adding a single bit to the state restores Markovianity completely.

Given an initial state $s_0$ and a rule for choosing actions, such as a policy $\pi(a_t \mid s_t)$, the chain rule factors the distribution over a full trajectory:

Reading this from right to left is exactly what a rollout does: sample an action, sample a next state, repeat. Reading it left to right is what a dataset of experience is: one or more draws from this trajectory distribution. This dual reading is worth internalizing, because it explains why training and evaluation can disagree so sharply. Training sees draws from the trajectory distribution generated by the data-collecting policy; evaluation asks the model to generate trajectories under a different policy, often one chosen by a planner that steers toward states the collecting policy rarely visited. The factorization is the same, but the regions of state space it explores are not.

The key quantitative phenomenon in rollouts is compounding error. Suppose your one-step model has a small error $\epsilon$ per step. If errors merely added, a hundred-step rollout would be off by $100\epsilon$. In reality they typically compound multiplicatively, because the model's predicted state at step $t$ is the input to its prediction at step $t+1$: mistakes change the inputs to future predictions, which changes the predictions, which changes the inputs further. The feedback loop is the decisive difference between open-loop fitting and closed-loop deployment, and no property of the training loss accounts for it. For stable systems the growth saturates; for unstable or chaotic ones it explodes exponentially. We will see both regimes concretely in the next section.

Let's watch uncertainty evolve under a simple stochastic transition, $s_{t+1} = 0.9\,s_t + w_t$ with $w_t \sim \mathcal{N}(0, 0.15^2)$, started from many identical initial conditions. Because every trajectory starts from exactly the same point, all spread in the ensemble is uncertainty the model itself would have to invent: none of it comes from the initial condition.

Prediction alone does not characterize a dynamical system. Ask where trajectories end up if you let them run, and you are asking about fixed points, stability, and attractors. These concepts are not optional decoration for world modelers: they determine whether a learned model's long-horizon rollouts look like the real system or like garbage, and they explain why one-step validation can be dangerously misleading. The shift in perspective is from local to global: a transition law tells you what happens in the next instant, while attractor structure tells you what happens in the long run, and the two can come apart in ways that matter enormously.

Fixed points matter because they organize the long-term behavior of everything around them. They act as the destinations toward which nearby trajectories drift or the sources from which they flee, and the inventory of a system's fixed points is therefore a rough sketch of its entire long-run personality. A trained world model whose rollouts converge to a spurious fixed point, for example a video model that predicts the blurry mean of all plausible futures, has failed in a specifically dynamical way, even if each individual one-step prediction looked plausible. The failure is invisible to one-step metrics because the spurious fixed point is approached gradually, one defensible step at a time.

Not all fixed points are equal. Some pull nearby trajectories in; others fling them away. The distinction is captured by stability. Consider a one-dimensional discrete-time system $x_{t+1} = f(x_t)$ with fixed point $x^*$. Nudge the state slightly: $x_t = x^* + \delta_t$. A Taylor expansion gives

so the perturbation evolves as $\delta_{t+1} \approx f'(x^*)\,\delta_t$. The nudge shrinks if $|f'(x^*)| < 1$ and grows if $|f'(x^*)| > 1$. Linearization therefore sorts fixed points into stable (perturbations decay) and unstable (perturbations grow) ones. The intuition is geometric: the derivative measures how strongly the map stretches or compresses small intervals, and repeated application multiplies that stretch factor, so anything less than one in magnitude decays geometrically while anything greater explodes geometrically.

The same idea in higher dimensions replaces the derivative with the Jacobian matrix $J = \partial f / \partial s$ evaluated at the fixed point, and the condition becomes: all eigenvalues of $J$ must lie strictly inside the unit circle of the complex plane. Each eigenvalue governs perturbations along its own eigendirection, so a system can be stable along one direction and unstable along another, which is why the worst eigenvalue decides. For continuous-time systems $\dot{s} = g(s)$, the condition is that all eigenvalues of the Jacobian have negative real part. If you take one piece of classical control mathematics into the rest of this handbook, make it this one, because it is exactly the diagnostic you will run, knowingly or not, whenever a learned model's rollouts blow up, oscillate, or collapse. A learned dynamics network is a function like any other; it has Jacobians at its fixed points, and its eigenvalues decide whether its imagined futures settle down or spiral apart.

Linearization is a local tool. It tells you what happens in an infinitesimal neighborhood of the fixed point and says nothing about which initial conditions eventually arrive there. That second question brings us to basins and attractors.

An attractor is a set that trajectories converge to and stay near. The main species, in increasing order of complexity:

• Fixed-point attractors: convergence to a single state, like a cup of coffee cooling to room temperature. All memory of the initial condition is eventually lost.
• Limit cycles: convergence to a repeating orbit, like a heartbeat or, as we will see shortly, a thermostat. The system forgets when it started but remembers a rhythm.
• Quasi-periodic tori: motion on two or more incommensurate frequencies, common in coupled oscillators. The motion never exactly repeats, yet remains regular and predictable over moderate horizons.
• Strange attractors: bounded but aperiodic behavior with sensitive dependence on initial conditions, the signature of chaos. The system stays in a recognizable region forever yet becomes unpredictable in detail.

Each attractor comes with a basin of attraction: the set of initial conditions that converge to it. Systems with several attractors partition their state space into basins, and which basin a trajectory starts in can matter more than any local detail of its motion. Basins introduce a discrete, qualitative sensitivity into otherwise smooth dynamics: nudge an initial condition across a basin boundary and its destiny changes completely, even though its immediate next steps barely change. Learned world models routinely distort basin boundaries, which is one reason a model can behave well in the data distribution yet fail catastrophically for the states a planner steers it toward. A planner that believes a boundary sits one place, when it actually sits somewhere else, will confidently steer across it and discover a different regime than its model promised.

The cleanest laboratory for these ideas is the logistic map,

a one-dimensional, deterministic, nonlinear system studied extensively by Robert May in the 1970s as a model of a self-limited population. Despite its simplicity, as the parameter $r$ increases it exhibits a fixed point, a cascade of periodic oscillations, and finally chaos. One quadratic term, one parameter, and every phenomenon in the attractor taxonomy above: this economy is why the map remains the standard first example four decades on.

We can analyze it exactly. The fixed points are $x^* = 0$ and $x^* = (r-1)/r$. Since $f'(x) = r(1 - 2x)$, the derivative at the nonzero fixed point is $f'(x^*) = 2 - r$, which is stable precisely when $|2 - r| < 1$, that is, for $1 < r < 3$. At $r = 3$ the fixed point loses stability and a stable period-two cycle is born. That cycle itself loses stability at $r = 1 + \sqrt{6} \approx 3.45$, giving way to period four, then eight, in a period-doubling cascade with remarkably universal spacing (Feigenbaum's constant $\delta \approx 4.669$), until chaos sets in near $r \approx 3.57$. Each doubling is a bifurcation: a stable attractor splits in two, and the system's long-run behavior acquires one more alternating phase.

There is a second lesson here for anyone training learned dynamics. A model fitted to minimize one-step squared error has no term in its objective that rewards getting the attractor right. It can achieve excellent one-step accuracy while its autonomous rollouts drift to a wrong fixed point, oscillate at the wrong frequency, or diverge entirely. The objective measures fidelity of the map; the deployment requirement is fidelity of the orbits, and the two coincide only when the map is accurate everywhere the orbit goes, which is precisely what a finite dataset cannot guarantee. We will demonstrate this failure concretely with the thermostat at the end of the chapter, and we will return to mitigation strategies, such as multi-step training objectives and rollout-based validation, repeatedly in Part VI and Part XI.

Probability tells us how to represent belief; dynamics tells us how belief evolves. Information theory tells us what is worth remembering. For world models this is not an abstract luxury: the central engineering question of the field is which aspects of a potentially infinite history must be carried forward in a finite state, and information theory is the discipline that answers such questions. Every architecture choice in latent world modeling, from bottleneck widths to memory mechanisms, is implicitly a bet about which information is dispensable, and the tools in this section make those bets explicit and checkable.

The entropy of a discrete random variable measures its uncertainty in bits:

A fair coin has $H = 1$ bit; a certain outcome has $H = 0$; a uniform draw from eight options has $H = 3$ bits. Entropy is also the answer to a compression question: $H(X)$ is the expected number of bits per symbol needed to encode draws from $X$ without loss, a result due to Shannon that anchors the deep connection between prediction and compression. Any system that predicts well can be turned into a compressor and vice versa, a duality that resurfaces in modern world models trained with likelihood objectives and in the coding-inspired analyses we will encounter in Part III. The intuition behind the coding theorem is worth holding onto: symbols you can predict need few bits, because the decoder can guess them, while surprising symbols need many. Prediction quality and code length are two views of the same quantity.

Conditioning reduces uncertainty on average. The conditional entropy $H(X \mid Y)$ measures the uncertainty remaining in $X$ after observing $Y$, and it satisfies $H(X \mid Y) \le H(X)$, with equality exactly when $X$ and $Y$ are independent. Observing something can never hurt, on average, though it can fail to help if it carries no relevant information. One technical caution: for continuous variables the direct analogue, differential entropy, can be negative and is not directly a bit count. In continuous settings, work with relative quantities such as mutual information, defined next, which behave cleanly.

Mutual information measures how much knowing one variable reduces uncertainty about another:

It is symmetric, nonnegative, and zero exactly when $X$ and $Y$ are independent. Unlike correlation, it detects nonlinear dependence: if $Y = X^2$, correlation is zero but mutual information is large. This matters for world models because the dependencies that carry predictive signal, resonances, thresholds, switching regimes, are typically nonlinear, and a team that audits its representations with correlations alone will certify as "uninformative" exactly the features doing the heavy lifting.

The single most consequential fact for our purposes is the data processing inequality: if $X$, $Y$, and $Z$ form a chain in which $Z$ is produced from $Y$ without further access to $X$, then

No transformation of the data, however clever, can increase the information it carries about the source. Deterministic processing can only preserve or destroy information. Stochastic processing destroys information in expectation as well, since randomness injected downstream cannot retroactively clarify the upstream signal. This innocent-looking inequality governs what encoders, bottlenecks, and abstractions are allowed to do, and it is the reason "compression" and "information loss" are precise engineering terms rather than metaphors. Whenever an encoder maps a rich observation history into a compact latent, the inequality certifies that whatever the latent fails to capture about the source is gone for good, no matter how deep the networks stacked on top of it.

Let's verify the inequality empirically on a simple communication channel: a source $X$ drawn uniformly from eight symbols passes through unchanged to $Y$, then degrades into $Z$ by additive noise.

The data processing inequality tells us compression loses information in general. Sufficiency characterizes the compression that loses nothing relevant.

The classic example: to estimate the mean of a Gaussian, the sample mean is sufficient; the individual orderings and deviations of the data points carry no additional information about that particular question. Sufficiency is always relative to a purpose. A summary sufficient for one question is usually wildly insufficient for another. The sample mean of a Gaussian tells you nothing about its variance; the running maximum of a river gauge is sufficient for flood risk but nearly useless for predicting next week's level. This relativity is not a weakness but the entire point: it converts the vague demand "keep everything important" into the precise demand "keep everything important for the specified question", and for world models the specified question is always prediction of the future.

Now assemble the pieces, because they snap together into the defining property of a good world-model state. Recall the Markov property from earlier: the future is conditionally independent of the past given the present state. Restate that in information-theoretic language:

The state $s_t$ is a sufficient statistic of the history for the purpose of predicting the future. Everything in the past that matters has been routed through the state; everything discarded was irrelevant. This single sentence connects three threads of this handbook:

• Filtering: when observations are partial, the belief state maintained by a Bayesian filter is designed to be exactly such a sufficient summary, and we will construct it in Part II: Inference, Decisions, and Control.
• State abstraction: when we search for compact states that preserve decision-relevant information while discarding the rest, we are searching for approximate sufficient statistics, the topic of Part III's treatment of state abstraction and sufficient statistics.
• Representation learning: when a latent-variable model is trained so that predictions depend on the past only through the latent, the data processing inequality certifies what the encoder is allowed to forget, a constraint we exploit in Part III's treatment of latent representations.

Approximation enters inevitably. Exact sufficiency usually requires carrying enormous state; practical world models trade a controlled amount of predictive information for a large reduction in state size. Much of the art of world modeling is choosing where along that trade-off curve to sit, and we will develop tools for judging that choice in Part XI. The thermostat below offers a rare treat: a system where the sufficient state is exactly one bit larger than the obvious choice, so we can see the cost of insufficiency in closed form rather than in vague degradation.

It is time to make the running example concrete. The system is a room with a heater, governed by a discrete-time transition:

where $T_t$ is room temperature, $m_t \in \{0, 1\}$ is the heater command, $\alpha = 0.6$ is the heating rate, $\kappa = 0.02$ is the cooling coefficient, $T_{\text{out}} = 8$ degrees is the outside temperature, and $\varepsilon_t$ is small thermal noise. The physics is deliberately minimal: linear heat flow with a constant outside sink, a constant heater input, and additive noise. Nothing here is hard to simulate, which is exactly why any modeling failure we observe later will be attributable to representation and dynamics rather than to simulation complexity.

The controller uses a hysteresis rule: turn the heater on when the temperature falls below 19.5 degrees, turn it off above 20.5 degrees, and otherwise keep the previous command.

Hysteresis is the interesting ingredient. Without it, the controller would flicker on and off at the threshold. With it, the controller's decision at the intermediate temperatures depends not only on the current temperature but on which side the system approached from, that is, on the past. Watch what this does to the dynamics.

This is the smallest possible instance of a problem that pervades the field. Partially observable systems hide state variables inside their histories, and agents that ignore the hidden component see a world that appears to violate the Markov property even when the underlying world is perfectly Markovian. The remedies, carrying beliefs over hidden states or learning memory-based representations, are major subjects of Part II and Part III. Here, we can simply measure the damage.

Let's play the role of a practitioner who has logged thermostat data and wants a world model. We will fit two one-step regression models to the same dataset: a full model that receives the current temperature and the heater command applied during that transition, and a baseline that receives temperature only. We will then evaluate both the way practitioners usually do, with one-step error, and the way that actually matters for a world model, by rolling each model out in closed loop and comparing against the true system's attractor. The experimental design is intentionally austere: linear models, least squares, no tuning. If the dynamical failure appears even here, it is not an artifact of deep learning.

First, build the training data and fit both models by least squares. We fit on the first 1,500 steps of the log and reserve the remaining 500 as held-out data, so the closed-loop stress test below is scored against temperatures the models never saw.

Read the figure alongside the printed numbers, because together they make the chapter's central practical argument. The full model, fitted with the command aligned to the transition it caused, reproduces the true system's limit-cycle regime. Its deterministic trajectory need not remain pointwise synchronized with a noisy held-out trajectory, so the terminal gap is not by itself a fidelity certificate. The baseline does noticeably worse on one-step error, but nothing about that error signals the scale of the failure to come: in closed loop, its distorted coefficients translate into a rollout that never oscillates at all. The fitted temperature coefficient, about 0.72, makes the learned map a contraction, so the rollout converges within roughly forty steps to a spurious fixed point near 20.03 degrees, comfortably inside the controller deadband, where the hysteresis rule keeps the heater permanently off. The model has averaged over the two heater regimes instead of representing them, and the result is systematic dynamical distortion: a constant erroneous temperature rather than the true limit cycle. A planner consuming this model would misjudge both how hot the room gets and how quickly it cycles, and it would do so with total confidence, because the model reports no uncertainty about its own distortion.

This is why the qualification test from Chapter 3 insists on separating open-loop prediction quality from closed-loop decision usefulness. A model can pass every one-step metric and still be useless, or actively harmful, as the substrate for planning. Whenever you train a world model, reserve part of your evaluation for autonomous rollouts compared against held-out true trajectories, and pay attention to whether the model reproduces the character of the system's attractors, not just its short-range accuracy. Concretely: compare oscillation periods, duty cycles, variance levels, and occupancy of state space, the statistical fingerprints introduced in the chaos discussion above. The thermostat needed only one hidden bit to break a naive pipeline; production environments hide far more, and far less legibly.

The key parameters for the transition-model experiment are:

• n_train: Number of logged steps used to fit both regression models. The rest of the log is held out, so closed-loop rollouts are compared against data the models never saw. Larger values reduce estimation noise in the fitted coefficients but shrink the held-out window available for the stress test, so the split embodies a direct trade-off between fit quality and evaluation power.
• rcond: Cutoff for small singular values passed to np.linalg.lstsq. Design directions whose singular values fall below this fraction of the largest are treated as zero, which keeps the least-squares solve stable when design columns are nearly collinear. In this experiment the design is benign, but in richer feature spaces the same knob controls how aggressively the solver refuses to fit noise directions.
• horizon: Number of closed-loop feedback steps in the stress-test rollout. Longer horizons give compounding error more room to separate a dynamically faithful model from one that is merely accurate one step ahead. Choose it relative to the system's natural timescale: here, a few dozen steps span several full limit cycles, which is enough for amplitude and rhythm errors to become unmistakable.
• temp0: Initial temperature handed to every closed-loop rollout. Matching it to the start of the true evaluation window keeps the gap measurements comparable across models. Because the system's limit cycle erases initial conditions within a handful of steps, this choice matters little for the full model, which is itself a small demonstration of attractor behavior.

The machinery in this chapter is powerful, but each piece carries assumptions that fail in instructive ways, and knowing the failure modes is as valuable as knowing the tools.

The Markov property is the assumption everything else rests on, and it is an assumption about representation, not about reality. Real environments routinely violate it for any state an agent can actually hold: other agents adapt, exogenous processes drift, and sensors alias distinct situations onto identical readings. When Markovianity fails, every technique built on it, from dynamic programming to latent rollouts, inherits a bias that no amount of data removes. More data collected under the same insufficient state reproduces the same aliasing more precisely, not less. The honest responses are to enlarge the state, to carry explicit beliefs over what is hidden, or to accept approximation and bound its cost, and each response defines a research lineage we follow in Part II and Part III.

Linearization, our main stability tool, is a local instrument applied to global questions. It correctly classifies hyperbolic fixed points (at non-hyperbolic equilibria the linear terms are inconclusive and higher-order analysis decides) but says nothing about basin boundaries, transient behavior far from equilibria, or systems whose interesting dynamics, like the logistic map's chaos, live nowhere near any fixed point. Estimating Jacobians and Lyapunov exponents from learned models compounds the model's own errors: a slightly biased model can report confidently that a fixed point is stable when it is not, or vice versa. Treat stability diagnostics of learned models as hypotheses to be checked by direct rollout experiments, not as certificates. The diagnostic and the rollout are complementary: the former explains a failure after the latter reveals it.

Information theory brings its own practical gaps. Entropy and mutual information are easy to define and notoriously hard to estimate in high dimensions; plug-in estimators are badly biased with finite samples, and the estimators that work well, such as variational bounds and contrastive methods, optimize surrogates whose tightness is rarely known. When this chapter says a good state satisfies $I(\text{future}; \text{past} \mid s_t) = 0$, treat it as a design ideal that guides architecture and evaluation, not as a quantity you will compute exactly outside of small discrete problems. The channel experiment above was trustworthy because eight symbols and two hundred thousand samples make estimation trivial; a latent of a few thousand dimensions offers no such comfort.

What the chapter provides, despite these caveats, is a shared language in which every subsequent debate becomes precise. Questions like "is this representation adequate?", "will rollouts diverge?", "how much memory does this task require?", and "is this model exploitable by a planner?" all reduce to statements about conditional distributions, transition stability, sufficient statistics, and compounding error. Practitioners who internalize this vocabulary consistently diagnose world-model failures faster, because most failures are instances of a handful of dynamical pathologies rather than novel mysteries.

• Random variables and conditional distributions are the substrate of world modeling. A world model is a stack of conditionals, and the chain rule composes them into full trajectory distributions. Conditioning both shifts and sharpens belief; models that report only conditional means discard the spread that planning needs.
• States, transitions, and trajectories formalize prediction. The transition kernel $p(s_{t+1} \mid s_t, a_t)$ generalizes the deterministic map, and the Markov property, an assumption about the state representation, licenses recurrences instead of ever-growing histories. Rollout uncertainty compounds, saturating for stable systems and exploding for unstable ones.
• Fixed points, stability, and attractors characterize long-run behavior. Linearization classifies fixed points via Jacobian eigenvalues, and attractors range from fixed points through limit cycles to chaos. Judge models of chaotic systems on statistical fidelity, not long-horizon point accuracy.
• Information, compression, and sufficient statistics say what a state must remember. The data processing inequality bounds what encoding can preserve, and a Markov state is exactly a sufficient statistic of the past for predicting the future, the ideal that state abstraction and latent representations approximate.
• The thermostat tied it all together: a limit-cycle attractor, a hidden one-bit state that temperature alone misses, and a vivid demonstration that one-step accuracy does not imply correct closed-loop dynamics.

With these foundations in place, the next chapter can do what this one deliberately deferred: attach actions, rewards, and optimality to the transition structure, arriving at the Markov decision process, the formal arena in which nearly all of Part II takes place.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the mathematical and dynamical foundations of world models.