Mathematical and Dynamical Foundations › States, Transitions, and Trajectories

With the probabilistic vocabulary in place, we can now define the object a world model maintains: a state that evolves through time according to a transition law, generating trajectories. The definitions in this section look elementary, but each one embeds a modeling decision that practitioners get wrong constantly, so we will be explicit about what each term commits us to.

Throughout this handbook we keep four concepts strictly separate, and it pays to fix them now:

• Observation: what the agent's sensors deliver at time $t$, written $o_t$. Observations may be incomplete, delayed, noisy, or redundant. A camera sees surfaces but not velocities; a thermometer reports temperature but not the heater's switch position.
• State: the information used to summarize the past for purposes of predicting the future, written $s_t$. The state may be observed directly, constructed by an encoder, or inferred as a belief. The state is defined by its predictive job, not by its physical interpretation.
• Action: the input the agent chooses, written $a_t$. Actions are the handle by which the agent influences the transition law, and their presence is what separates prediction from control.
• Environment: the external process that consumes $(s_t, a_t)$ or $(o_t, a_t)$ and produces the next observation, along with any reward. The environment is the ground truth against which every model in this handbook is ultimately judged.

A state is good not because it resembles the world internally but because it is useful for prediction. This pragmatic attitude, which goes back to the cybernetic tradition discussed in Chapter 5, lets us sidestep philosophical questions about what the "real" state of the world is. A thermostat's state can be two numbers and a switch position; it does not need to contain the position of every air molecule in the room. The molecular detail is real, but it is irrelevant to the predictive question the controller faces, and carrying it would be pure overhead. This criterion, usefulness for prediction, is the thread that connects the state concept to the sufficient statistics of the final section: a good state is precisely a summary that keeps what matters and discards what does not.

A transition law specifies how the state evolves. In the simplest case it is a deterministic function:

$$s_{t+1} = f(s_t, a_t).$$

Deterministic transitions are an idealization. Real systems are buffeted by unmodeled influences: sensor noise, discretization error, and genuine randomness. The general treatment is a transition kernel, a conditional distribution over next states:

$$s_{t+1} \sim p(\cdot \mid s_t, a_t).$$

The deterministic case is the special case where this distribution puts all its mass on one point. A useful mental model is that $f(s_t, a_t)$ predicts the center of the next-state distribution while the kernel also carries its spread. Models that track only the center, which is what most neural network dynamics models do by default, silently assume the spread does not matter. Over long horizons, it always does: spread is what tells a planner whether a risky maneuver has a five percent or a fifty percent chance of failure, and a model blind to spread answers that question with confident nonsense.

The Markov property deserves emphasis because it is the hinge between this chapter and nearly everything that follows. It is what allows a world model to be a recurrence rather than a function of ever-growing histories, and it is what makes dynamic programming, filtering, and reinforcement learning tractable at all. Without it, every prediction problem would require re-processing the entire history from scratch, and the computational cost of planning would grow without bound as experience accumulates. With it, the entire past is compressed into a fixed-size object, and all reasoning proceeds forward from there. When we reach Part II: Inference, Decisions, and Control, the Markov decision process will formalize exactly this structure with actions and rewards attached. When the state you have is not Markov, the fixes are either to enrich the state, which is the subject of state abstraction in Part III, or to carry an explicit belief over hidden states, which is the subject of Bayesian filtering. The thermostat later in this chapter gives a concrete taste of the first remedy: adding a single bit to the state restores Markovianity completely.

Given an initial state $s_0$ and a rule for choosing actions, such as a policy $\pi(a_t \mid s_t)$, the chain rule factors the distribution over a full trajectory:

$$p(s_{0:T}, a_{0:T}) = p(s_0) \prod_{t=0}^{T-1} \pi(a_t \mid s_t)\, p(s_{t+1} \mid s_t, a_t).$$

Reading this from right to left is exactly what a rollout does: sample an action, sample a next state, repeat. Reading it left to right is what a dataset of experience is: one or more draws from this trajectory distribution. This dual reading is worth internalizing, because it explains why training and evaluation can disagree so sharply. Training sees draws from the trajectory distribution generated by the data-collecting policy; evaluation asks the model to generate trajectories under a different policy, often one chosen by a planner that steers toward states the collecting policy rarely visited. The factorization is the same, but the regions of state space it explores are not.

The key quantitative phenomenon in rollouts is compounding error. Suppose your one-step model has a small error $\epsilon$ per step. If errors merely added, a hundred-step rollout would be off by $100\epsilon$. In reality they typically compound multiplicatively, because the model's predicted state at step $t$ is the input to its prediction at step $t+1$: mistakes change the inputs to future predictions, which changes the predictions, which changes the inputs further. The feedback loop is the decisive difference between open-loop fitting and closed-loop deployment, and no property of the training loss accounts for it. For stable systems the growth saturates; for unstable or chaotic ones it explodes exponentially. We will see both regimes concretely in the next section.

Let's watch uncertainty evolve under a simple stochastic transition, $s_{t+1} = 0.9\,s_t + w_t$ with $w_t \sim \mathcal{N}(0, 0.15^2)$, started from many identical initial conditions. Because every trajectory starts from exactly the same point, all spread in the ensemble is uncertainty the model itself would have to invent: none of it comes from the initial condition.