Markov and Partially Observable Decision Processes › From Prediction to Decisions

This section connects the dynamical systems view from Part I to the decision-making view we need now, and fixes the vocabulary we will use for the rest of the book.

Recall from Mathematical and Dynamical Foundations that a controlled dynamical system evolves according to a state transition rule such as $s_{t+1} = f(s_t, a_t, w_t)$, where $a_t$ is a control input and $w_t$ is noise. A world model, as we framed it in What Is a World Model?, is essentially a learned version of $f$: given the current situation and a candidate action, it predicts what happens next. Nothing in that formulation says whether the action is good, bad, or indifferent. The model is a neutral simulator, and if you hand an agent nothing but a simulator, the best it can do is enumerate futures, not rank them.

Decision making adds a scoring layer. We assume the environment emits a scalar reward after each transition, and the agent's goal is to collect as much cumulative reward as possible over time. This single assumption is remarkably productive: navigation, game playing, resource management, and motor control can all be cast as reward accumulation, and the rest of this chapter is about reasoning precisely with it. It is worth pausing on how strong a simplification this is. Every objective we might care about, whether it is "deliver the package," "minimize energy," or "keep the factory running," gets flattened onto a single number line. That flattening buys enormous mathematical power: comparing any two behavior patterns reduces to comparing two numbers. It also carries risk, since a poorly chosen reward measures something other than what we intended. We return to that danger in the Limitations section; for now, note that the entire machinery ahead takes the reward function as given and asks only how to maximize it.

Because the words state, observation, and action will carry heavy loads, let us fix their meanings now:

• State ($s$): the situation of the environment, including anything that influences its future. The agent may or may not have access to it.
• Observation ($o$): what the agent actually senses. Observations can be incomplete, noisy, or ambiguous about the state.
• Action ($a$): a decision the agent selects, which influences the transition of the state.
• Transition ($P$): the probabilistic rule by which states evolve under actions.
• Reward ($r$): a scalar signal emitted by the environment, treated as given rather than chosen by the agent.
• Policy ($\pi$): the agent's decision rule, mapping whatever it knows to an action.
• Environment: the process that consumes actions and produces next states, observations, and rewards.

Keeping these roles distinct matters more than it might seem. Much confusion in applied work comes from blurring state with observation, or from treating a learned latent representation as if it were the environment state. For example, when a neural network compresses camera frames into a latent vector, it is tempting to call that vector "the state." Whether it deserves the name depends entirely on whether it preserves everything relevant to the future, which is precisely the question the Markov property will let us pose precisely. We will return to that distinction repeatedly.