Control as Inference and Active Inference › Active Inference

Active inference adopts the same mathematical substrate, generative models, approximate posteriors, free-energy minimization, but assembles them into a description of an organism embedded in a sensorimotor loop, rather than a planner solving a specified task. Where control as inference starts from an engineer's MDP and adds probabilistic packaging, active inference starts from a cognitive-systems premise: a living thing is, above all, a predictor of its own sensations, and everything else it does serves that predictive mandate. Its distinctive commitments are threefold, and each one changes the engineering in ways you can trace line by line through implementations.

• Preferred observations, not externally assigned rewards. The agent carries a prior $p(o)$ over what it expects or likes to sense. Deviation between predicted sensations and preferred sensations plays the reward's role, and an internal scalar objective is recovered by setting $r(o) \propto \log p_{\text{pref}}(o)$. This is a subtle reframing with real consequences. An engineered reward is assigned from outside the agent; a preference is represented inside its generative model. Preferences can be learned, genetically encoded, or culturally transmitted, and because they live in the same probability calculus as everything else, they compose naturally with uncertainty over what the world will deliver.
• Whole policies as random variables. Many active-inference formulations introduce an explicit random variable over complete candidate policies $\pi$, rather than representing control only through the state-conditioned action distributions produced by backward messages. This is a difference in parameterization and unit of choice, not a claim that control-as-inference posteriors never range over complete trajectories. Evaluating a policy means imagining the observations it would elicit, which is exactly what a world model enables. Deliberation therefore looks like mental simulation: roll out the consequences of a whole course of action in imagination, score the simulated sensations, then compare scores across candidates before committing to even the first move.
• Model parameters participate. The hidden things beliefs must capture include not only latent state but the parameters $\theta$ of the model itself. Acting can improve your model, and that possibility belongs inside the objective rather than in a bolt-on curiosity module. This commitment makes ignorance-of-the-model a first-class citizen of the theory: an agent that knows its own transition table is incomplete behaves differently from one that does not, and the difference emerges from the mathematics rather than from hand-coded meta-reasoning.

Perception and action then unify into a single imperative: minimize free energy over posterior beliefs, using whichever instrument is available, revising beliefs about state, revising $\theta$ through learning, or moving the body to gather confirming sensation. On this view there is no fundamental line between perception (changing beliefs to fit the world) and action (changing the world to fit beliefs); they are two directions of the same error-correction process, and which one fires depends on which is cheaper or more reliable in the moment. Friston's slogan for this is self-evidencing: organisms select the sensory evidence their own models predict. Critics hear circularity there, an agent destined to confirm its own prejudices by seeking only what it expects; defenders reply that prediction-error minimization plus preference satisfaction is precisely what survival demands, since an organism that persistently encounters surprising sensation is an organism whose model of the world is failing it. The theory's devotees span neuroscience, psychology, and a growing machine-learning engineering community around tools such as pymdp.

The working heart of discrete-time active inference is a formula for grading candidate policies before committing to them. Suppose the agent holds approximate beliefs over current state and possibly over parameters. For a candidate $\pi$, the model predicts a distribution over the next observation $o'$ and the state $s'$ that would generate it. One widely used single-step decomposition of expected free energy is

$$G(\pi) = \underbrace{ D_{\mathrm{KL}}\!\big(q(o'\mid\pi)\,\big\Vert\,p_{\mathrm{pref}}(o')\big) }_{\text{risk: predicted mismatch with preferences}} \;+\; \underbrace{ \mathbb{E}_{q(s' \mid \pi)} \left[ \mathcal{H}\big(p(o' \mid s')\big) \right]}_{\text{ambiguity: how noisy the channel is}} .$$

Under the usual predictive factorization, this same quantity can be written as preference cross-entropy minus information gain about hidden state,

$$G(\pi) = -\mathbb{E}_{q(o'\mid\pi)}[\log p_{\mathrm{pref}}(o')] - I_q(s';o'\mid\pi).$$

Thus the risk-plus-ambiguity form already contains a hidden-state epistemic-value term. When model parameters $\theta$ are uncertain, many implementations extend the objective with a separate novelty term, subtracting the expected information gain $I_q(\theta;o'\mid\pi)$. Over a multi-step policy, the corresponding per-step quantities are summed across the planning horizon. This extended form is the one relevant to the parameter-learning toy below.

• Risk compares the full predicted outcome distribution with the preferred outcome distribution. Minimizing it favors predictions concentrated where preferences place probability mass. For finite-width sharp preferences this has the familiar negative-log-preference cost shape; in a literal point-mass limit it diverges unless the prediction also becomes a matching point mass.
• Ambiguity penalizes situations where even knowing the state leaves the observation unpredictable, dim sensors, unreliable actuators. If the true observation channel has aleatoric noise, that irreducible floor cannot be learned away. The agent's estimated channel entropy can still change as its model is learned or its belief over states is refined; averaged over uncertain states, the term therefore favors places where the current model expects sensing to work, an idea that echoes the observability concerns of Part II, Chapter 2 pushed to a decision-making context.
• Epistemic value rewards observations expected to reduce hidden-state uncertainty; it appears explicitly in the cross-entropy-minus-information-gain form above. Expected information gain about uncertain parameters is often called novelty. Novelty is the quantity Bayesian experimental design maximizes, and operationally it is curiosity about the model: an agent can be certain about where it is and yet deeply uncertain about what happens when it pushes the strange button.

An agent minimizing $G$ therefore pursues a bundle of aims simultaneously. Where classical control supplies one number per trajectory, expected free energy supplies a decomposable profile: how much this act pleases the agent, how trustworthy the sensing is along the way, and what the agent stands to learn about hidden states and, in the extended objective, model parameters. The decomposition is diagnostic gold in engineered systems, because a deployed agent behaving strangely can be interrogated term by term: is it chasing a mis-specified preference, fleeing ambiguous sensing, or chasing information at a compulsive rate? A prior over policies can additionally bias posterior policy selection toward habituated courses of action; Part VII revisits deliberation-versus-habit arbitration.

Two structural remarks connect this back across the book. First, in linear-Gaussian problems with Gaussian preferences, the risk term reproduces quadratic tracking costs up to optimization-invariant constants when the relevant predicted covariances do not depend on the policy. If they do, the Gaussian KL also contributes policy-dependent covariance terms: a trace penalty together with an entropy-like log-determinant term, jointly favoring covariance that matches the preference distribution rather than simply driving variance to zero. Connections to Kalman estimation remain natural, but an exact identification of ambiguity with estimation cost requires additional assumptions, because additive observation noise can make ambiguity policy-independent. Second, when $\theta$ is known, parameter novelty vanishes, but hidden-state epistemic value need not. Only under additional conditions that make the information-gain and ambiguity contributions policy-independent does minimizing expected free energy reduce to maximizing expected shaped reward $r(o)=\log p_{\mathrm{pref}}(o)$. Entropy-regularized or risk-sensitive policy optima also depend on the policy-distribution or trajectory-level free-energy term. The two traditions therefore overlap without being algebraically identical in every specialization, a point made carefully in comparative analyses by Millidge, Tschantz, and Buckley and by Lanillos and colleagues.

Because readers meet both traditions in the wild, it pays to see the correspondences laid side by side.

As Table framework-comparison shows, reading down the middle columns reveals the deepest alignment: both frameworks replace argmax control with an evidence-weighted, entropy-respecting posterior over behavior, and both use the same softmax mathematics. Reading the outer columns reveals substantive divergences. Control as inference treats the reward as god-given and exploration as thermal noise, indifferent randomness spread evenly over suboptimal options; active inference treats preferences as introspective expectations and exploration as inquiry, targeted at exactly the places where the model admits it is weakest. That difference is vivid in practice: crank up a Boltzmann temperature and the agent wanders everywhere a little more, including places it knows perfectly well; raise the weight on salience and it beelines toward specific unknowns. Whether an engineered system needs the richer apparatus is a modeling judgment, not a theorem, and the worked examples below are designed to help you exercise that judgment on toy problems where both accounts can be computed exactly and their disagreement made visible.

Before the laboratory, let us honor the arc of the book. Chapter 1 of this part gave us MDPs, Chapter 2 gave us beliefs over latent state, Chapter 4 gave us Bellman optimality. This chapter folded all three into one probabilistic object: state estimation that feeds a generative model, decisions read off a posterior, and dynamic programming reborn as message passing. What remains classical is the hard max. Everything ahead, value learning under function approximation, planning in imagination, hierarchical agency, modulates or exploits softness in some way. The softness you watched emerge from exponential tilting today is the same softness that lets tomorrow's algorithms hedge against networks that hallucinate transitions with perfect confidence.