Control as Inference and Active Inference › Limitations and Practical Impact

The strongest objection to control as inference is foundational. Calling $\exp(r/\beta)$ a "likelihood of optimality" requires care: as written it is an unnormalized weight, not probability mass, and a measure-theoretic likelihood interpretation would require an explicit base measure and normalization. The evidence term $p(\mathcal{O})$ is a valid partition function, but it need not represent the physical frequency of an optimality event. A further concern follows directly in stochastic-reward settings: two equal-return trajectories receive the same exponential tilt, so that factor alone cannot distinguish skill from luck; their full posterior weights may still differ through the base plausibility $p(\tau)$. A learner that interprets the tilt as competence can therefore make a category error. In practice this means you should not lean on the posterior's absolute normalizer and should test the induced ranking under stochastic outcomes.

Practical limitations cluster in familiar territory. The posterior absorbs whatever biases live in the model, and since real world models are learned and wrong, the exponential tilt that makes plans sharp also makes confident nonsense sharp. A model that mispredicts door locations will produce a gorgeously concentrated route posterior aimed at a wall, a Goodhart dynamic we return to in Part XI: Evaluation and Understanding and in the reliability chapters of Part XII. Softening helps but is not a cure: a high-temperature agent wastes effort gracefully instead of wasting it confidently, and neither mode discovers the model is lying. Scale is a second constraint: exact message passing grows with horizon, state cardinality, and action branching, while exhaustive horizon-$T$ deliberation over deterministic action sequences requires $|\mathcal A|^T$ candidates and up to $T$ imagined transitions per candidate. Engineered systems therefore approximate aggressively, often collapsing trajectory posteriors into value functions or amortized policies. Whole-candidate-policy active-inference schemes inherit the same combinatorial pressure; how far current continuous-domain implementations have progressed toward production depends on the application and is an empirical systems question, not a mathematical conclusion established here. Preference specification is the third sore spot: active inference advertises freedom from externally imposed rewards, but someone must write down $p(o)$, the prior over preferred observations. In high-dimensional or correlated observation spaces, specifying a full coherent distribution can be harder than assigning a scalar reward. The freedom is real, but so is the displacement of the design burden: nothing is eliminated, only relocated somewhere that feels more biological and may be less convenient to engineer.

Against these liabilities stands a substantial practical legacy. The identification of planning with inference gave the field a common mathematical language connecting control theory, probabilistic programming, and deep generative models: maximum-entropy reinforcement learning, path-integral planners, and maximum-entropy inverse reinforcement learning all use pieces of this chapter's mathematics. Active inference contributes a complementary normative story in which perception, learning, and action share one objective. Both traditions show that stochastic behavior is not necessarily sloppiness; it can be the deliberate consequence of an entropy-regularized or information-seeking objective, and distinguishing those cases is a core competency when evaluating learned agents.

When you meet an implemented system claiming these labels, interrogate it along lines the chapter has equipped you to use:

• Recovery test: as temperature approaches $0^+$, or preference sharpness increases, does behavior approach the classical optimum? Failure warrants an audit of inference error, finite-horizon truncation, sampling noise, ties, and model mismatch; it does not diagnose lossy inference by itself.
• Calibration: do the posterior's predicted outcome distributions match empirical frequencies along held-out rollouts?
• Exploration quality: is information seeking directed at actual uncertainty reduction, or merely thermal noise wearing a Bayesian costume? Compare against an ablation with the salience term removed.
• Robustness under model error: degrade the generative model deliberately and measure how gracefully the soft policy degrades compared with its hard-max twin.
• Preference-reward conversion: if the system switches between reward and preference descriptions, do the two produce consistent rankings of the same candidate policies?