Control as Inference and Active Inference › Summary

• Control problems can be restated as inference over behavior: augment a bare generative model with binary optimality factors tied to reward through $p(\mathcal{O}_t{=}1) \propto \exp(r/\beta)$, using a state-action-independent omitted multiplier, and posterior conditioning gives $p(\tau \mid \mathcal{O}_{1:T}) \propto p(\tau)\, e^{R(\tau)/\beta}$ after those multipliers are absorbed into the global normalizer. This identity is the finite-horizon undiscounted construction; discounting requires time-indexed factors $e^{\gamma^{t-1}r_t/\beta}$, while the implementation uses the related discounted maximum-entropy Bellman surrogate.
• Exact backward messages in the augmented finite-horizon graph use a log-of-expectation state update. A related maximum-entropy Bellman surrogate replaces the hard action maximum with temperature-controlled log-sum-exp and induces a Boltzmann policy over soft Q-values.
• The same recursions optimize the maximum-entropy control objective, equipping policies with hedging behavior that hedges model error, a property model-based RL leans on heavily, as the next chapter explores in depth.
• When exact posteriors are unattainable, free energy supplies the universal variational currency: minimize $\mathcal{F}(q) = \mathbb{E}_q[-\log p(\text{model}, \text{evidence})] - \mathcal{H}[q]$ over tractable $q$.
• Active inference extends the currency prospectively: candidate policies are graded by expected free energy. Its risk-plus-ambiguity decomposition is equivalent to preference cross-entropy minus hidden-state information gain; uncertain-parameter formulations can add a novelty term for expected information gain about model parameters.
• The frameworks agree on softness, share softmax mathematics, and differ on where objectives come from, whether agents know their own models are incomplete, and how the exchange rate between knowing and getting is fixed. These are engineering degrees of freedom, and exercising them well is a recurring judgment call in the chapters ahead.

You now hold the tool that unifies the first five chapters of this part: one grammar in which filters, planners, and controllers are all posterior machines. Next we put pressure on that grammar. In Model-Based Reinforcement Learning, the generative model stops being known and becomes something learned from data, and we discover what remains of optimality when the evidence is produced by a network that can be confidently, catastrophically wrong.