Markov and Partially Observable Decision Processes › Limitations and Impact

The MDP and POMDP formalisms give control, operations research, and reinforcement learning a shared vocabulary for dynamics, observations, objectives, and decisions. When we speak of a world model as something an agent plans against, the mathematical content of that sentence is an approximation of $P$, $R$, and, under partial observability, $O$, evaluated through the value-function lens this chapter built. Learned latent-state systems preserve the same division of labor when an encoder supplies an internal state, a transition model predicts how it evolves, and a policy or planner chooses actions against those predictions.

The limitations are equally structural, and it is better to meet them now than to discover them mid-project. First, the curse of dimensionality is not an implementation detail but a property of the representation. A belief over $|\mathcal{S}|$ enumerated states lies on a simplex of dimension $|\mathcal{S}|-1$; if the underlying state combines $m$ variables with $d$ possibilities each, then $|\mathcal{S}|=d^m$ and the belief representation grows exponentially in $m$. This arithmetic alone explains why exact tabular methods quickly become impractical and why later chapters rely on sampled trajectories, learned value functions, point-based belief planning, or amortized policies. Second, the framework takes the reward function as given, yet specifying rewards that capture what we actually want is notoriously difficult, and optimizing a misspecified reward faithfully can be worse than useless. This is the scalar-reward flattening danger promised in the opening: a single number can collapse incompatible goals, hide distributional consequences, and reward proxies that cease to track the intended outcome. An agent that maximizes the letter of a flawed specification does so relentlessly; the scalar objective treated here as an axiom is also a major source of silent failure. The safety questions of Part XII begin right here, at the moment we declare a scalar sufficient. Third, the assumptions are fragile in practice. Stationarity fails as mechanisms wear and other agents adapt, while the Markov property fails whenever the chosen representation loses information. These are distinct failure modes, though predictable drift can sometimes be absorbed into an enlarged state. As the T-maze showed, representation loss can flip a profitable agent into a losing one. The engineering responses (enriching the state, inferring beliefs, learning representations that preserve decision-relevant information, and adapting the dynamics model under drift) are not optional refinements. They are where most of the real difficulty lives.

A final limitation deserves emphasis because it recurs throughout the book: everything in this chapter evaluates a policy against a model of the environment, and the model can be wrong. Value iteration on an inaccurate $P$ produces confidently wrong values, and an agent that plans against its own model can exploit model errors in ways that look brilliant in simulation and fail in reality. Notice the asymmetry between the two kinds of error: errors in $P$ distort predictions broadly, but errors concentrated along the trajectory the plan happens to prefer are the dangerous ones, because the optimizer actively seeks out whichever futures the model rates most favorably, accurate or not. This gap between model-space optimality and environment-space performance, sometimes called model exploitation, is a first-class concern we will carry into model-based reinforcement learning and the reliability chapters. The Monte Carlo validation loop from the gridworld experiment, comparing planned values against realized returns, is the humble prototype of the safeguards those chapters develop.