Markov and Partially Observable Decision Processes › Summary

This chapter built the formal backbone for everything that follows:

• An MDP $(\mathcal{S}, \mathcal{A}, P, R, \gamma, \rho_0)$ couples a world model (the transition kernel) with an objective (the reward function) and a time preference (the discount factor).
• The Markov property licenses compressing entire histories into the current state. It is a property of a representation, not of the world, and it can fail through sensor aliasing, omitted latent variables, observation delays, exogenous processes, or lossy abstraction. Nonstationarity is a separate dynamics assumption: under drift, the same represented state and action need not retain the same transition law over time.
• Returns accumulate discounted reward; the effective horizon $1/(1-\gamma)$ expresses how far ahead the agent reasons.
• Value functions convert long-horizon expectations into per-state scores. The Bellman equations make them recursive, and the Bellman operator's $\gamma$-contraction guarantees that value iteration converges to the unique optimal value function, from which a greedy policy is recovered.
• POMDPs model partial observability with an emission model. The optimal policy depends on history, but the belief state, the posterior over true states given history, restores the Markov property at the cost of a continuous, high-dimensional state space.
• The T-maze experiment quantified the stakes: maintaining a belief recovered most of an oracle's performance, while reactive policies lost reward on average. Memory is not a convenience; under partial observability it is the difference between profit and loss.

You now have the decision-theoretic skeleton on which every later chapter hangs. You can define a decision problem precisely, evaluate any policy against it recursively, plan optimally when the model is known and the state visible, and state exactly which assumptions collapse when observation becomes incomplete. The immediate next step is the one this chapter motivated but deferred: how to compute and maintain beliefs efficiently, with the Kalman filter as the canonical case. That is the subject of the next chapter, Bayesian Filtering and Belief States.