Mathematical and Dynamical Foundations › Limitations and Impact

The machinery in this chapter is powerful, but each piece carries assumptions that fail in instructive ways, and knowing the failure modes is as valuable as knowing the tools.

The Markov property is the assumption everything else rests on, and it is an assumption about representation, not about reality. Real environments routinely violate it for any state an agent can actually hold: other agents adapt, exogenous processes drift, and sensors alias distinct situations onto identical readings. When Markovianity fails, every technique built on it, from dynamic programming to latent rollouts, inherits a bias that no amount of data removes. More data collected under the same insufficient state reproduces the same aliasing more precisely, not less. The honest responses are to enlarge the state, to carry explicit beliefs over what is hidden, or to accept approximation and bound its cost, and each response defines a research lineage we follow in Part II and Part III.

Linearization, our main stability tool, is a local instrument applied to global questions. It correctly classifies hyperbolic fixed points (at non-hyperbolic equilibria the linear terms are inconclusive and higher-order analysis decides) but says nothing about basin boundaries, transient behavior far from equilibria, or systems whose interesting dynamics, like the logistic map's chaos, live nowhere near any fixed point. Estimating Jacobians and Lyapunov exponents from learned models compounds the model's own errors: a slightly biased model can report confidently that a fixed point is stable when it is not, or vice versa. Treat stability diagnostics of learned models as hypotheses to be checked by direct rollout experiments, not as certificates. The diagnostic and the rollout are complementary: the former explains a failure after the latter reveals it.

Information theory brings its own practical gaps. Entropy and mutual information are easy to define and notoriously hard to estimate in high dimensions; plug-in estimators are badly biased with finite samples, and the estimators that work well, such as variational bounds and contrastive methods, optimize surrogates whose tightness is rarely known. When this chapter says a good state satisfies $I(\text{future}; \text{past} \mid s_t) = 0$, treat it as a design ideal that guides architecture and evaluation, not as a quantity you will compute exactly outside of small discrete problems. The channel experiment above was trustworthy because eight symbols and two hundred thousand samples make estimation trivial; a latent of a few thousand dimensions offers no such comfort.

What the chapter provides, despite these caveats, is a shared language in which every subsequent debate becomes precise. Questions like "is this representation adequate?", "will rollouts diverge?", "how much memory does this task require?", and "is this model exploitable by a planner?" all reduce to statements about conditional distributions, transition stability, sufficient statistics, and compounding error. Practitioners who internalize this vocabulary consistently diagnose world-model failures faster, because most failures are instances of a handful of dynamical pathologies rather than novel mysteries.