Control as Inference and Active Inference

Imagine a warehouse robot that starts in a corner and must reach a charging dock. The machinery of Classical Planning and Optimal Control gives us one way to think about it: define a cost, solve for the cost-to-go, and act greedily by minimizing the one-step cost plus the successor's cost-to-go. This chapter asks a different, almost inverted question. Instead of asking "what is the best action?", we ask "what would an optimal controller have done here, and can we infer it from evidence?" When we phrase control as a problem of reasoning under uncertainty, something surprising happens: planning, value iteration, exploration, and even the way humans appear to make decisions all fall out of a single probabilistic machine.

It is worth dwelling on how radical this inversion is before we build it. Classical control treats optimality as a goal to be achieved: the agent searches a space of policies and commits to the best one it can find. The inference view treats optimality as an observation to be explained: the agent imagines that the actions it is about to take were generated by something near-optimal, and then works backward from that assumption to recover what those actions must have been. The first stance is that of an engineer; the second is that of a statistician reconstructing causes from effects. What makes the inversion powerful is that the reconstruction problem comes with a deep toolkit: Bayes' rule going back centuries, followed much more recently by message passing and variational approximation. We do not have to invent control algorithms from scratch; we can adapt mature inference machinery.

Two closely related traditions developed this idea. Control as inference, rooted in work by Attias, Toussaint and Storkey, Ziebart, Todorov, and Kappen, and synthesized in Sergey Levine's 2018 tutorial, embeds an artificial optimality variable into a generative model and computes the posterior over trajectories conditioned on optimality. Active inference, developed by Karl Friston, Thomas Parr, Giovanni Pezzulo, and many others in the theoretical neuroscience tradition, reaches a strikingly similar mathematical structure from a different premise: agents minimize a quantity called free energy because they must maintain a coherent model of their own sensory flux. Both traditions replace the hard maximum of classical control with a softer, evidence-weighted form of reasoning, and both inherit consequences that matter enormously for world models: natural stochasticity in behavior, principled exploration, and a delicate dependence on the quality of the underlying generative model.

This material supports several later connections. The Model-Based Reinforcement Learning chapter carries the optimality-variable construction forward when it develops maximum-entropy methods, while its planning-in-imagination algorithms draw more broadly on model-based planning. Part VII returns to information-seeking behavior and the value of actions that improve an agent's model. The central point is this: the hard max that has anchored every optimal-control equation so far is about to become optional, and the way we pay for that softening determines the resulting policy.

Recall the picture from Classical Planning and Optimal Control: a Markov decision process with dynamics $p(s_{t+1} \mid s_t, a_t)$ and a cost or reward signal, solved by dynamic programming to produce a policy $\pi(a \mid s)$ that maximizes expected discounted return. The solution is prescriptive. It tells you what to do. The Bellman equation is a command: at each state, take the best action, where "best" is defined with all the force of an argmax operator. There is no room in that formalism for hesitation between two nearly equivalent routes, no place to record that three different behaviors were all pretty good, and no mechanism by which acting stochastically could ever be rational if acting deterministically is available.

Control as inference is descriptive first and prescriptive second. We start with bare dynamics and no special status for actions: a trajectory $\tau = (s_1, a_1, s_2, a_2, \ldots, s_T)$ is simply drawn from a generative model in which states evolve and actions occur according to some vague prior. Then we bolt on one extra piece of evidence: a binary variable $\mathcal{O}_t$ for each timestep that says whether time $t$ was "optimal". Conditioning on the observation that every timestep was optimal converts the generative model into a distribution over near-optimal behavior. Solving the control problem now looks like a familiar statistical task: compute (or approximate) a posterior. Notice what has happened to the direction of reasoning: instead of pushing preferences forward through time to decide what to do, we assume good behavior exists and reason backward to discover what it looks like.

Why go through this indirection? It would be fair to object that we are making life complicated on purpose: we had a working optimization framework, and we wrapped it in probabilistic clothing. The probabilistic formulation adds capabilities the bare optimization framework structurally lacked. Three reasons recur throughout the literature.

• Planning becomes inference tooling. Any approximate posterior machinery, message passing, variational inference, Monte Carlo sampling, becomes a candidate planner. This is a substantial practical dividend. The community has spent decades building fast, scalable, well-understood inference algorithms for graphical models; the moment planning is phrased as posterior computation, that entire library of machinery applies directly to decision-making without modification.
• Exploration becomes automatic. The posterior over optimal trajectories is multimodal: it spreads mass over all behaviors that plausibly earned high return, rather than collapsing onto a single deterministic winner. We will see that this yields maximum-entropy reinforcement learning. Nobody writes "add noise for exploration" into the algorithm; the hedging is intrinsic to conditioning on soft evidence, and its magnitude is set by a single interpretable dial.
• Agents that reason about their own models get a unified currency. As we saw in Bayesian Filtering and Belief States, perception is already posterior inference over latent state. Control as inference makes action the same kind of computation over the same kind of object. Perception and action stop being two modules glued together and become two uses of one calculus, which is precisely the unification that makes world-model-based agents conceptually coherent.

Let us make the construction precise. Fix a finite horizon $T$. The generative model over trajectories has the usual chain structure, with one addition per timestep: an optimality node $\mathcal{O}_t$ that depends on $(s_t, a_t)$ and is treated as observed. This dependence structure matters: $\mathcal{O}_t$ hangs off the state-action pair like a sensor reading, since optimality is judged by what happened at that step alone, mediated through nothing else in the graph.

Read the graph left to right: latent states chain forward, actions are drawn from a (typically uniform) prior, and each pair of state and action emits evidence about its own optimality. Nothing in this graph knows about rewards yet. It describes a random system bumbling through states taking arbitrary actions, exactly the setup from which HMM smoothing begins. The reward enters through one modeling choice, a single sentence linking evidence to reward:

Here $\propto$ denotes an unnormalized factor: the omitted multiplier is independent of $(s_t,a_t)$ at each timestep. That convention is what lets the product of omitted multipliers be absorbed into one trajectory-level normalizer. The function $r$ is the familiar reward and $\beta > 0$ is a temperature: small $\beta$ means optimality evidence is sharp and demanding, while large $\beta$ tolerates sloppy, high-entropy behavior. Equivalently, $1/\beta$ is the inverse temperature. The terminology comes from statistical physics, where temperature governs how strongly a Boltzmann distribution concentrates on low-energy configurations; here it plays the same role with reward standing in for negative energy. The message-passing section below shows how the exponential form permits backward recursions and how a common maximum-entropy relaxation restores the Bellman machinery from the previous chapter.

Conditioning on all optimality nodes gives the core object of the whole framework:

This is worth pausing on, because the entire chapter lives inside this equation. Under the unnormalized-factor convention above, all state-action-independent multipliers are absorbed into $p(\mathcal{O}_{1:T})$, leaving the dynamics prior tilted by the exponential of trajectory return. High-return trajectories occupy most of the posterior mass; mediocre ones retain some; terrible ones are exponentially suppressed but never eliminated. Because exponential weighting never assigns exactly zero probability to anything, even catastrophically bad trajectories survive with vanishing-but-positive weight, which will matter when we discuss exploration and robustness later. The sum in the exponent is the plain additive return from Part II Chapter 1, which is precisely why reward and optimality evidence can be identified without changing the underlying decision problem. Had we chosen any other encoding, the quantity being traded off across timesteps would not be return, and we would have silently invented a different objective.

One honesty checkpoint before proceeding. Treating an event like "this trajectory had return $R$" as having probability $\exp(R/\beta)$ stretches the word probability. As written, the exponential is an unnormalized weight: it is not probability mass, and calling it a likelihood would require specifying a base measure and normalization. (A continuous density may exceed one, so magnitude alone is not the objection.) The clean return tilt also relies on the omitted factor being independent of state and action; normalizing the binary conditional separately at every $(s,a)$ would generally introduce state-action-dependent constants and change the posterior. The partition function $p(\mathcal{O}_{1:T})$ is mathematically well defined, but it need not be the physical chance of witnessing a repeatable optimality event. The construction survives because we use relative weights within the posterior, and those ratios remain well defined regardless of whether the absolute scale has a physical interpretation. The practical boundary is therefore simple: treat the exponential as an unnormalized tilt, not as a calibrated physical probability, and test the induced ranking under the stochastic outcomes that matter.

Computing the full trajectory posterior by enumerating trajectories is hopeless, since the number of length-$T$ trajectories grows exponentially in both horizon and branching factor; even our humble $5 \times 5$ grid admits $4^{140} \approx 1.9 \times 10^{84}$ action sequences within the rollout horizons we will sample shortly. But we do not need to enumerate anything. A backward message produces the causally conditioned action distribution $p(a_t \mid s_t, \mathcal{O}_{t:T})$ used for control. Full smoothing marginals under $\mathcal{O}_{1:T}$ combine that backward message with a forward message, exactly as HMM smoothing combines both directions without enumerating state sequences. The remarkable fact is what the backward messages turn out to be.

Define dimensionless backward log-messages $\bar Q_t(s,a)=\log p(\mathcal{O}_{t:T}\mid s_t{=}s,a_t{=}a)$ and $\bar V_t(s)=\log\sum_a p(a)\exp(\bar Q_t(s,a))$. Log-space converts products of evidence into sums. For stochastic transitions, exact marginalization gives

The logarithm sits outside the next-state expectation. Replacing this risk-sensitive state aggregation with a linear expectation yields the familiar entropy-regularized Bellman operator used by maximum-entropy reinforcement learning,

When $\gamma=1$, replacing the log-of-expectation state update by a linear expectation is exact for deterministic transitions. The stationary operator displayed here uses $\gamma<1$ later in the grid world, so even with deterministic dynamics it should be read as the fixed point of a discounted maximum-entropy control objective, not as the exact backward-message recursion of the finite optimality graph. Its action aggregation is an actual log-sum-exp, so as $\beta\to0^+$ the induced policy concentrates on hard-max actions; as $\beta$ grows, inferior actions receive more mass. Encoding a discounted return directly in the optimality graph would instead use time-dependent evidence factors such as $\exp(\gamma^{t-1}r_t/\beta)$; the resulting exact recursion carries that time dependence and is not the stationary $\gamma$-on-value operator above.

We can watch this interpolation happen directly. Take four illustrative action values with the spread produced by our grid world, compute the implied Boltzmann weights across a range of temperatures, and plot each weight against its value: as $\beta$ shrinks, mass collapses onto the best action; as it grows, all actions converge toward equal weight.

At $\beta = 0.05$ the top-valued action holds nearly every unit of weight, the visual signature of a recovered Bellman backup. At $\beta = 2.0$ the four actions sit almost shoulder to shoulder near one quarter each, evidence that has nearly stopped constraining anything.

Discounting by $\gamma$ mirrors how classical control discounts future rewards; in the purely probabilistic reading, finite horizons induce similar decay, and $\gamma$ can be seen as either a modeling choice about patience or an analytic convenience for keeping infinite-horizon sums finite.

To connect the message notation to the usual reward-scale action value, define $Q_t(s,a)=\beta\bar Q_t(s,a)$. The action prior remains part of the marginal:

For the uniform action prior used throughout the worked example, $p(a)$ is the same for every action and cancels during normalization, leaving the familiar softmax $\pi(a\mid s)\propto\exp(Q(s,a)/\beta)$. The demo_Q values above are on this reward scale, so softmax_weights(demo_Q, beta) divides by temperature exactly once.

This is the celebrated Boltzmann (softmax) rationality rule. The little equation is one of the most reused artifacts in modern decision-making research, and it is worth appreciating just how far it travels. It appears as an assumption about human irrationality in behavioral economics, where noisy choices are modeled as deterministic preferences plus thermal perturbation; as the choice model underpinning inverse reinforcement learning, where recovering rewards from demonstrations requires assuming something about how demonstrators soften their own optimality; as the temperature in language-model sampling, where the same dial trades fluency against diversity; and as the exploration rule in dozens of RL algorithms. Here we derived it; nowhere did we tell the agent to explore stochastically. Stochasticity falls out of the posterior, with probability mass allocated across actions in exact proportion to how plausible each is under the optimality hypothesis.

Backward message passing in the finite chain is exact when it retains the log-of-expectation state update above. The stationary linear-expectation recursion is a maximum-entropy control relaxation that can be run as value iteration with a softened action maximum. Planning and belief propagation therefore share a common algebraic template, but they are not interchangeable under every stochastic model. Keeping that boundary explicit lets us borrow inference intuitions without claiming exact marginals where the implemented operator computes a control surrogate.

This equivalence is not merely elegant. It matters practically because a maximum-entropy policy can act as a distributed hedge: it keeps mass on alternative routes, so ordinary replanning has options available after a model error. That is a useful design intuition, not a universal robustness guarantee; a soft policy can still spread probability onto dangerous actions. The warehouse experiment's Temperature Dial below measures this instance directly with seeded closed-loop rollouts, plotting mean return and policy entropy across seven temperatures. The observed direction is an empirical property of this setup, not a theorem for every environment.

Everything so far assumed the trajectory posterior is computable. The finite tabular chain with known dynamics and one observed optimality factor per timestep retains a local factorization, so forward-backward message passing can compute its marginals exactly. What fails in realistic world models, with continuous state, partial observability, or learned nontrivial dynamics, is usually closed-form evaluation of the required sums or integrals, not the chain factorization itself. A different construction with one unobserved global tilt shared across timesteps can also destroy the per-step factorization. Our toy grid world uses a discounted maximum-entropy Bellman surrogate and samples its induced policy rather than drawing exact finite-horizon posterior trajectories. For these less tractable settings we approximate, and the canonical approximation device is variational inference, already your acquaintance from latent-variable modeling.

Pick a tractable family of distributions $q(\tau)$ and minimize the gap between $q$ and the true posterior. That gap is measured by a KL divergence we cannot evaluate directly, since evaluating it would require knowing the very posterior we are trying to approximate, but it is sandwiched by a quantity we can:

$\mathcal{F}(q)$ is the (negative) evidence lower bound, called free energy for reasons inherited from statistical physics: it trades off expected "energy" (how well $q$ fits the model and evidence) against entropy (how much $q$ refuses to commit). The physics analogy is more than decoration. In thermodynamics, systems at fixed temperature settle into states balancing energetic preference against entropic dispersion, and Helmholtz free energy measures exactly that compromise; variational inference borrows the name because it minimizes the same structural tradeoff, with $q$ playing the role of the system's configuration distribution. Minimizing $\mathcal{F}$ both tightens the bound on evidence and drives $q$ toward the true posterior; equality holds exactly when $q$ equals the posterior, in which case the bound saturates and the gap vanishes identically.

To make the tradeoff concrete, consider a miniature world whose true posterior over two outcomes is known, then let $q$ be a Bernoulli family indexed by its own probability parameter. Sweeping that parameter traces the full free-energy curve: its cross-entropy term falls as $q$ shifts mass toward the evidence-favored outcome, its negative-entropy term resists complete commitment, and their sum reaches its minimum precisely at the true posterior.

The dashed cross-entropy term alone is linear in $q$ and would drive it to the boundary $q=1$, placing all mass on the more probable outcome. The dotted negative-entropy term resists that complete commitment; their sum bottoms out at $q=0.75$, exactly where $q$ reproduces the true posterior. That is the graphical meaning of a saturated evidence bound. Readers who have met variational posteriors will recognize the shape immediately, since the same curve governs every ELBO ever maximized. Written out fully for trajectories, the quantity being minimized is

Every ingredient here is a friend from earlier chapters. The energy term scores proposed trajectories by exactly the generative model a world model provides; the entropy term preserves the hedging spirit of the previous section, ensuring the optimum does not collapse onto a single point estimate of the future. Because this machinery makes contact with your study of variational latent-variable models, note the kinship with the ELBO: same inequality, different cast of characters. There the evidence was a datapoint and $q$ approximated a posterior over latents; here the evidence is "the trajectory was near-optimal" and $q$ approximates a posterior over complete behaviors. The algebra does not notice the difference.

Applied to control specifically, the idea becomes powerful when the variational family $q$ is itself a set of feedback policies: if $q(\tau)$ factors as the law of the state process under a time-varying policy $\pi_t(a \mid s)$, minimizing free energy selects the best policy available in that family. This is optimization through an inference objective, not a guarantee of monotone Bellman policy improvement. An implementation may avoid writing an explicit value table, but the exact messages still satisfy the same soft Bellman structure derived above; the computational recursion has been reinterpreted, not abolished. Related inference and path-integral formulations yield tractable risk-sensitive linear-quadratic special cases, showing that the lens is not confined to grids and tables. In continuous spaces with Gaussian dynamics it can produce weighted-trajectory formulas; in learned models, trained as in Part V and Part VI, it becomes an optimizer over imagined rollouts, grading candidate futures against the generative model before any action touches the real world.

Keep one asymmetry in mind, because it is the hinge on which the next section swings. Free energy $\mathcal{F}$ is retrospective: it scores beliefs about trajectories that include evidence you have. It answers "how well did my beliefs explain what I observed?", a question about the past and present. An agent that must commit to future actions needs a prospective cousin, a quantity that scores proposed courses of action before the evidence arrives, averaging over the observations they would produce. That cousin, the expected free energy, is where active inference lives.

Active inference adopts the same mathematical substrate, generative models, approximate posteriors, free-energy minimization, but assembles them into a description of an organism embedded in a sensorimotor loop, rather than a planner solving a specified task. Where control as inference starts from an engineer's MDP and adds probabilistic packaging, active inference starts from a cognitive-systems premise: a living thing is, above all, a predictor of its own sensations, and everything else it does serves that predictive mandate. Its distinctive commitments are threefold, and each one changes the engineering in ways you can trace line by line through implementations.

• Preferred observations, not externally assigned rewards. The agent carries a prior $p(o)$ over what it expects or likes to sense. Deviation between predicted sensations and preferred sensations plays the reward's role, and an internal scalar objective is recovered by setting $r(o) \propto \log p_{\text{pref}}(o)$. This is a subtle reframing with real consequences. An engineered reward is assigned from outside the agent; a preference is represented inside its generative model. Preferences can be learned, genetically encoded, or culturally transmitted, and because they live in the same probability calculus as everything else, they compose naturally with uncertainty over what the world will deliver.
• Whole policies as random variables. Many active-inference formulations introduce an explicit random variable over complete candidate policies $\pi$, rather than representing control only through the state-conditioned action distributions produced by backward messages. This is a difference in parameterization and unit of choice, not a claim that control-as-inference posteriors never range over complete trajectories. Evaluating a policy means imagining the observations it would elicit, which is exactly what a world model enables. Deliberation therefore looks like mental simulation: roll out the consequences of a whole course of action in imagination, score the simulated sensations, then compare scores across candidates before committing to even the first move.
• Model parameters participate. The hidden things beliefs must capture include not only latent state but the parameters $\theta$ of the model itself. Acting can improve your model, and that possibility belongs inside the objective rather than in a bolt-on curiosity module. This commitment makes ignorance-of-the-model a first-class citizen of the theory: an agent that knows its own transition table is incomplete behaves differently from one that does not, and the difference emerges from the mathematics rather than from hand-coded meta-reasoning.

Perception and action then unify into a single imperative: minimize free energy over posterior beliefs, using whichever instrument is available, revising beliefs about state, revising $\theta$ through learning, or moving the body to gather confirming sensation. On this view there is no fundamental line between perception (changing beliefs to fit the world) and action (changing the world to fit beliefs); they are two directions of the same error-correction process, and which one fires depends on which is cheaper or more reliable in the moment. Friston's slogan for this is self-evidencing: organisms select the sensory evidence their own models predict. Critics hear circularity there, an agent destined to confirm its own prejudices by seeking only what it expects; defenders reply that prediction-error minimization plus preference satisfaction is precisely what survival demands, since an organism that persistently encounters surprising sensation is an organism whose model of the world is failing it. The theory's devotees span neuroscience, psychology, and a growing machine-learning engineering community around tools such as pymdp.

The working heart of discrete-time active inference is a formula for grading candidate policies before committing to them. Suppose the agent holds approximate beliefs over current state and possibly over parameters. For a candidate $\pi$, the model predicts a distribution over the next observation $o'$ and the state $s'$ that would generate it. One widely used single-step decomposition of expected free energy is

Under the usual predictive factorization, this same quantity can be written as preference cross-entropy minus information gain about hidden state,

Thus the risk-plus-ambiguity form already contains a hidden-state epistemic-value term. When model parameters $\theta$ are uncertain, many implementations extend the objective with a separate novelty term, subtracting the expected information gain $I_q(\theta;o'\mid\pi)$. Over a multi-step policy, the corresponding per-step quantities are summed across the planning horizon. This extended form is the one relevant to the parameter-learning toy below.

• Risk compares the full predicted outcome distribution with the preferred outcome distribution. Minimizing it favors predictions concentrated where preferences place probability mass. For finite-width sharp preferences this has the familiar negative-log-preference cost shape; in a literal point-mass limit it diverges unless the prediction also becomes a matching point mass.
• Ambiguity penalizes situations where even knowing the state leaves the observation unpredictable, dim sensors, unreliable actuators. If the true observation channel has aleatoric noise, that irreducible floor cannot be learned away. The agent's estimated channel entropy can still change as its model is learned or its belief over states is refined; averaged over uncertain states, the term therefore favors places where the current model expects sensing to work, an idea that echoes the observability concerns of Part II, Chapter 2 pushed to a decision-making context.
• Epistemic value rewards observations expected to reduce hidden-state uncertainty; it appears explicitly in the cross-entropy-minus-information-gain form above. Expected information gain about uncertain parameters is often called novelty. Novelty is the quantity Bayesian experimental design maximizes, and operationally it is curiosity about the model: an agent can be certain about where it is and yet deeply uncertain about what happens when it pushes the strange button.

An agent minimizing $G$ therefore pursues a bundle of aims simultaneously. Where classical control supplies one number per trajectory, expected free energy supplies a decomposable profile: how much this act pleases the agent, how trustworthy the sensing is along the way, and what the agent stands to learn about hidden states and, in the extended objective, model parameters. The decomposition is diagnostic gold in engineered systems, because a deployed agent behaving strangely can be interrogated term by term: is it chasing a mis-specified preference, fleeing ambiguous sensing, or chasing information at a compulsive rate? A prior over policies can additionally bias posterior policy selection toward habituated courses of action; Part VII revisits deliberation-versus-habit arbitration.

Two structural remarks connect this back across the book. First, in linear-Gaussian problems with Gaussian preferences, the risk term reproduces quadratic tracking costs up to optimization-invariant constants when the relevant predicted covariances do not depend on the policy. If they do, the Gaussian KL also contributes policy-dependent covariance terms: a trace penalty together with an entropy-like log-determinant term, jointly favoring covariance that matches the preference distribution rather than simply driving variance to zero. Connections to Kalman estimation remain natural, but an exact identification of ambiguity with estimation cost requires additional assumptions, because additive observation noise can make ambiguity policy-independent. Second, when $\theta$ is known, parameter novelty vanishes, but hidden-state epistemic value need not. Only under additional conditions that make the information-gain and ambiguity contributions policy-independent does minimizing expected free energy reduce to maximizing expected shaped reward $r(o)=\log p_{\mathrm{pref}}(o)$. Entropy-regularized or risk-sensitive policy optima also depend on the policy-distribution or trajectory-level free-energy term. The two traditions therefore overlap without being algebraically identical in every specialization, a point made carefully in comparative analyses by Millidge, Tschantz, and Buckley and by Lanillos and colleagues.

Because readers meet both traditions in the wild, it pays to see the correspondences laid side by side.

As Table framework-comparison shows, reading down the middle columns reveals the deepest alignment: both frameworks replace argmax control with an evidence-weighted, entropy-respecting posterior over behavior, and both use the same softmax mathematics. Reading the outer columns reveals substantive divergences. Control as inference treats the reward as god-given and exploration as thermal noise, indifferent randomness spread evenly over suboptimal options; active inference treats preferences as introspective expectations and exploration as inquiry, targeted at exactly the places where the model admits it is weakest. That difference is vivid in practice: crank up a Boltzmann temperature and the agent wanders everywhere a little more, including places it knows perfectly well; raise the weight on salience and it beelines toward specific unknowns. Whether an engineered system needs the richer apparatus is a modeling judgment, not a theorem, and the worked examples below are designed to help you exercise that judgment on toy problems where both accounts can be computed exactly and their disagreement made visible.

Before the laboratory, let us honor the arc of the book. Chapter 1 of this part gave us MDPs, Chapter 2 gave us beliefs over latent state, Chapter 4 gave us Bellman optimality. This chapter folded all three into one probabilistic object: state estimation that feeds a generative model, decisions read off a posterior, and dynamic programming reborn as message passing. What remains classical is the hard max. Everything ahead, value learning under function approximation, planning in imagination, hierarchical agency, modulates or exploits softness in some way. The softness you watched emerge from exponential tilting today is the same softness that lets tomorrow's algorithms hedge against networks that hallucinate transitions with perfect confidence.

Enough abstraction. Let us compute a control problem entirely by inference and inspect what the machinery actually produces, so that every symbol from the derivations acquires a concrete address on a map.

Consider a $5 \times 5$ warehouse. Our courier robot starts at the bottom-left tile; the charging dock sits at the top-right. Four moves exist: up, right, down, left. Motors are imperfect: with probability $0.15$ the commanded direction is replaced by a direction sampled uniformly from all four moves. The replacement can therefore coincide with the original command; at walls, outward directions are clamped to a no-motion outcome. Each occupied moment costs $0.2$ (time is money), docking grants $10$, and the discounted maximum-entropy solver below uses $\gamma=0.96$. This is the Bellman surrogate described above, not the undiscounted finite-horizon posterior identity. An exact discounted trajectory tilt would instead use time-indexed evidence factors $\exp(\gamma^{t-1}r_t/\beta)$. Boundary clamping makes a wall-adjacent position potentially more robust because an outward replacement becomes a no-motion outcome rather than a lateral displacement, but the computed policy must determine whether that benefit outweighs the remaining chance of losing progress. The mirror-symmetric geometry makes the upward-first and rightward-first route families equivalent.

The inference view tells a richer story. We declare that the robot's actions were generated by some near-optimal strategy, express that declaration as optimality evidence $p(\mathcal{O}_t{=}1 \mid s, a) \propto \exp(r/\beta)$, and compute the posterior over trajectories. What should we expect? The answer depends entirely on where we set the dial, and the three regimes are worth naming before we compute:

• With small $\beta$, optimality evidence is draconian. Only near-perfect routes survive conditioning, and the trajectory posterior concentrates on the family of shortest routes. In this symmetric grid, the eight moves contain four upward and four rightward commands, giving $\binom{8}{4}=70$ monotone routes. Under the implemented replacement model, each commanded step reaches its intended neighbor with probability $0.85 + 0.15/4 = 0.8875$, so every specified monotone state sequence has the same dynamics-prior weight, $0.8875^8 \approx 0.385$. By contrast, $0.85^8 \approx 0.272$ is the probability that no replacement event occurs at all. Wall adjacency matters to robustness under alternative slip outcomes, not to the relative weight of those already-specified shortest trajectories. The exponential tilt suppresses routes that forfeit even a few points of return, so effectively only the winners remain visible.
• With large $\beta$, evidence barely constrains behavior. The posterior spreads diffusely over the whole grid: wanderings, detours, loops, all faintly tinged with the hint that docking is nice. Nothing is forbidden; trajectories are merely graded, gently.
• Between the extremes lies the interesting regime: a corridor of plausibility around the direct route, wide enough to admit multiple near-equivalent paths. Here the posterior encodes something no argmax policy can represent: both the best plan and a calibrated census of all plans that were close contenders.

Inference thereby replaces a single prescribed route with a calibrated probability distribution over routes, and the temperature dial $\beta$ dictates its sharpness. The implementation section turns this narrative into running code and measures the tradeoff the dial governs: as $\beta$ rises, behavior becomes more exploratory and the expected return decays along a measurable curve.

The same example lets us preview active inference's distinguishing trait. Suppose the warehouse contains a door whose opening mechanism the robot's model has never verified. At sufficiently low temperature, a reward-only control-as-inference courier strongly suppresses the costly detour and heads toward the dock. At higher temperature it may sample the door incidentally, but its exploration remains undirected because no information-seeking term values the uncertainty reduction itself. An expected-free-energy courier assigns salience to the mystery. Even at a slight cost in immediate preference satisfaction, peeking at the mechanism can slash its uncertainty and change everything about later decisions, especially if it turns out the door is a shortcut. The second half of the implementation constructs the smallest possible problem exhibiting exactly this contrast and quantifies the trade.

We will build the grid-world generative model, run soft Bellman iteration, sample trajectories from the induced policy, sweep the temperature, and finish with the toy salience computation. Every quantity is computed on the fly, and each figure renders from precomputed values only. The correspondence to keep in mind as the code unfolds: transitions is the generative model's dynamics, soft_value_iteration is the discounted maximum-entropy control surrogate motivated by backward message passing, and the softmax policy approximates the corresponding optimality-conditioned action marginal rather than reproducing the exact finite-horizon posterior.

First the environment: transition tensor, reward vector, start and goal.

Each row of transitions[s, a] is a probability vector, so one matrix-vector product computes expectations over successors. That single property makes the soft Bellman update a one-liner shortly, and it is the reason dense tensor representations remain useful for small discrete worlds even when fancier structures exist. Note the absorbing construction at the goal: the planner explicitly zeros the goal state's immediate self-loop reward, while the simulator enforces the same terminal convention by breaking as soon as the sampled rollout reaches the dock. Both pieces of terminal bookkeeping are necessary to prevent repeated docking rewards.

Now the planner. Dividing rewards by the temperature $\beta$ creates dimensionless soft values when $\beta$ is measured in reward units per nat. This implements the linear-expectation maximum-entropy Bellman surrogate distinguished from exact stochastic message passing above.

Three readings confirm the theory. Arrows along the top and right walls are bold: the posterior strongly favors dockward moves without ever applying an explicit maximum. Arrows in the interior are lighter: several directions compete there because slipping sideways costs little distance, so multiple actions carry comparable posterior credibility rather than a single winner with token runner-ups. Values rise through the approach states, but the starred absorbing dock is a local dip: its self-loop earns no further reward while the log-sum-exp action aggregation retains an entropy baseline. With four equivalent self-loop actions, the fixed point is $V(\text{dock})=\log 4/(1-\gamma)\approx34.66$ in reward-over-temperature units, independent of $\beta$ under this convention—not zero. Because predecessor actions differ in their probability of reaching the dock, this pinned value can also change their soft action probabilities. The heat map and policy should therefore be interpreted under this explicit terminal convention. Darker arrows carry stronger posterior preference; lighter ones mark states where several actions remain credible.

Exact ancestral sampling from the full trajectory posterior requires a forward pass over the grid accumulating posterior weight followed by a backward pass drawing states, standard HMM smoothing transplanted to this graph: the forward-backward algorithm computes filtering and smoothing messages just as it did for hidden states, with optimality evidence playing the role that observation emissions played before. A useful, much simpler surrogate is to roll out the Markov policy induced by the soft Bellman fixed point in the true environment. Those trajectories approximate $p(\tau \mid \mathcal{O})$; they are not exact ancestral samples of it, because the update above uses a discounted linear expectation over stochastic successors rather than exact log-message passing over a fixed horizon. For a five-by-five grid the surrogate is informative, but the shortcut should be used with care on longer horizons or more stochastic dynamics. We roll 400 simulated shifts per temperature and tally which tiles got visited.

At $\beta = 0.1$ the route posterior leans toward the short-wall routes, but it is far from a ribbon: with slips dispersing probability uniformly in every direction, roughly 40% of visits land in the interior and all four sides of the perimeter carry comparable traffic. At $\beta = 2.0$ the ribbon dissolves into mist; the warmest cell is the start corner, where steps accumulate before diffuse wandering reaches the dock, whose tile has relatively low occupancy because episodes terminate there. Neither picture is "correct". They are answers to different questions about how good a shift needs to be before we call it optimal, and choosing $\beta$ is choosing that question. This is perhaps the most practically important sentence in the worked example: practitioners who treat the temperature as a nuisance hyperparameter miss that it encodes an explicit stance on how much performance must be sacrificed for coverage, a stance better made deliberately than defaulted into.

Having characterized two endpoints, let us measure the full tradeoff curve that the dial governs.

The two panels trace one phenomenon from opposite sides. Return erodes steadily as $\beta$ climbs because diffuse posteriors waste steps; entropy climbs because soft posteriors hedge. Read together, they price the exchange: each nat of behavioral diversity costs a measurable slice of return, and the curve tells you the going rate in this warehouse. There is no universally best dial setting. A deployed fleet wants low temperature, since its job is delivery, not data collection; an agent collecting training data, or facing a world model whose errors it has not yet discovered, wants headroom, and the entropy regularization view makes that headroom a hyperparameter with measurable consequences. Next chapter's model-based RL algorithms inherit exactly this knob, dressed as the entropy coefficient of soft actor-critic, and the curves above are precisely the sort of diagnostic used to tune it.

Finally, the promised glimpse of salience, the one ingredient temperature-based schemes lack. Consider a two-layer mystery. Treat the unknown valve configuration as a two-valued model parameter $\theta$: under $\theta_0$, curtain 0's valve succeeds with probability $0.9$ (and curtain 1's with $0.1$); under $\theta_1$ the roles reverse. The robot assigns equal prior probability to $\theta_0$ and $\theta_1$, so the two curtains are tied initially; Strategy A breaks that tie by choosing curtain 0. One optional inspection gesture reveals the parameter instantly, at the cost of consuming that turn, which would otherwise have been a chance to activate a valve. Successes matter later, and each activated turn's outcome can itself shift beliefs slightly, which is why Strategy A gains a little information incidentally as a byproduct of acting.

We score two candidate strategies with a transparent surrogate inspired by two expected-free-energy components: expected negative log likelihood of preferred success outcomes and expected information gain about the unknown model parameter $\theta$. The toy assumes the inspection itself is unambiguous and does not claim to reproduce the full risk-plus-ambiguity decomposition above.

Strategy A accumulates substantial expected surprise from an uninformed first activation while harvesting information incidentally from both outcomes. Strategy B converts one sacrificial turn into a full bit of certainty. Its lower expected surprise already makes its expected free energy smaller; the taller information-gain bar widens that margin even though the reward ledger prefers A.

Read the discrepancy between the last lines slowly, because it is the entire philosophical difference between the two frameworks compressed into one toy. Strategy A, staying home and firing valves, collects more expected reward ($1.32$ versus $0.90$). Yet expected free energy crowns Strategy B for two reasons: its expected surprise is already lower, largely because Strategy A's first activation is an uninformed 50/50 choice, and perfect inspection contributes a full bit of certainty ($\log 2 \approx 0.69$ nats), more than the information in Strategy A's complete adaptive outcome history. Strategy A does react to its first outcome and therefore selects the better curtain on the second turn; its cost is the uninformed first attempt, not two consecutive choices of the wrong curtain. Strategy B's informed activation achieves the ceiling allowed by physics, since even perfect knowledge leaves the ten-percent failure rate untouched. In the surrogate above, surprises are treated as additive across the turn structure, a deliberate simplification for the sake of transparency, but the selection outcome is robust to those details.

Does this make active inference right and reward maximization naive? No. It makes them disagree on the exchange rate between knowledge and satisfaction. Reinforcement learning prices information explicitly: an inspection is worth it exactly when later decisions improve enough to repay the lost turn, and with a long horizon, Strategy A's follow-up gains close most of the gap, a calculation explored deeply in Model-Based Reinforcement Learning and in Part VII. Active inference fixes the exchange rate internally through the units of free energy itself, which yields sweeping, seemingly gratuitous curiosity in unfamiliar worlds and persistent sampling even at stable optima. Which behavior you want is a design decision about the deployment environment, not a theorem. A factory floor punishes gratuitous curiosity; a research probe earns its budget with it.

The key parameters for planning as message passing are:

• GAMMA: Discount factor multiplying future soft values during iteration. Values closer to 1 weight eventual docking more heavily against accumulated step costs, stretching the corridor of viable routes outward; smaller values myopically favor early escape from step costs and can shorten imagined plans accordingly.
• beta: Temperature dividing rewards inside the soft Bellman backup; $1/\beta$ is the inverse temperature. Small values sharpen optimality evidence toward the deterministic plan, while large values soften the posterior toward exploration.
• SLIP: Probability that the commanded move is replaced by a uniformly sampled direction, including the commanded direction itself. Raising it thickens the interior wall buffers of the value field and, at fixed $\beta$, shifts more posterior mass toward wall-adjacent routes where outward replacements are partially neutralized by boundary clamping.
• STEP_COST, GOAL_REWARD: Per-timestep cost and docking bonus that define the reward signal converted into optimality evidence. Their magnitudes relative to $\beta$, together with $\gamma$ and the horizon, set the effective urgency of arrival. Scaling every reward by $c$ while also scaling the temperature by $c$ leaves $r/\beta$ unchanged, so only reward-to-temperature ratios are policy-relevant under this parameterization.
• max_iter: Cap on fixed-point iterations, paired with a relative stopping tolerance of delta < 1e-7 scaled by the value magnitude. Convergence is geometric thanks to the contraction property shared with the hard Bellman operator, so the cap rarely binds on problems of this size.
• n_ep, max_steps: Number of sampled rollouts per temperature (400) and the per-rollout horizon cap (140 steps) used to estimate route density and mean return. The horizon cap bounds runtime but truncates episodes that never reach the dock. Because diffuse high-$\beta$ policies wander longer, they are the most exposed to truncation; increase the cap and rerun a sensitivity check before interpreting their occupancy fields or mean returns.

The strongest objection to control as inference is foundational. Calling $\exp(r/\beta)$ a "likelihood of optimality" requires care: as written it is an unnormalized weight, not probability mass, and a measure-theoretic likelihood interpretation would require an explicit base measure and normalization. The evidence term $p(\mathcal{O})$ is a valid partition function, but it need not represent the physical frequency of an optimality event. A further concern follows directly in stochastic-reward settings: two equal-return trajectories receive the same exponential tilt, so that factor alone cannot distinguish skill from luck; their full posterior weights may still differ through the base plausibility $p(\tau)$. A learner that interprets the tilt as competence can therefore make a category error. In practice this means you should not lean on the posterior's absolute normalizer and should test the induced ranking under stochastic outcomes.

Practical limitations cluster in familiar territory. The posterior absorbs whatever biases live in the model, and since real world models are learned and wrong, the exponential tilt that makes plans sharp also makes confident nonsense sharp. A model that mispredicts door locations will produce a gorgeously concentrated route posterior aimed at a wall, a Goodhart dynamic we return to in Part XI: Evaluation and Understanding and in the reliability chapters of Part XII. Softening helps but is not a cure: a high-temperature agent wastes effort gracefully instead of wasting it confidently, and neither mode discovers the model is lying. Scale is a second constraint: exact message passing grows with horizon, state cardinality, and action branching, while exhaustive horizon-$T$ deliberation over deterministic action sequences requires $|\mathcal A|^T$ candidates and up to $T$ imagined transitions per candidate. Engineered systems therefore approximate aggressively, often collapsing trajectory posteriors into value functions or amortized policies. Whole-candidate-policy active-inference schemes inherit the same combinatorial pressure; how far current continuous-domain implementations have progressed toward production depends on the application and is an empirical systems question, not a mathematical conclusion established here. Preference specification is the third sore spot: active inference advertises freedom from externally imposed rewards, but someone must write down $p(o)$, the prior over preferred observations. In high-dimensional or correlated observation spaces, specifying a full coherent distribution can be harder than assigning a scalar reward. The freedom is real, but so is the displacement of the design burden: nothing is eliminated, only relocated somewhere that feels more biological and may be less convenient to engineer.

Against these liabilities stands a substantial practical legacy. The identification of planning with inference gave the field a common mathematical language connecting control theory, probabilistic programming, and deep generative models: maximum-entropy reinforcement learning, path-integral planners, and maximum-entropy inverse reinforcement learning all use pieces of this chapter's mathematics. Active inference contributes a complementary normative story in which perception, learning, and action share one objective. Both traditions show that stochastic behavior is not necessarily sloppiness; it can be the deliberate consequence of an entropy-regularized or information-seeking objective, and distinguishing those cases is a core competency when evaluating learned agents.

When you meet an implemented system claiming these labels, interrogate it along lines the chapter has equipped you to use:

• Recovery test: as temperature approaches $0^+$, or preference sharpness increases, does behavior approach the classical optimum? Failure warrants an audit of inference error, finite-horizon truncation, sampling noise, ties, and model mismatch; it does not diagnose lossy inference by itself.
• Calibration: do the posterior's predicted outcome distributions match empirical frequencies along held-out rollouts?
• Exploration quality: is information seeking directed at actual uncertainty reduction, or merely thermal noise wearing a Bayesian costume? Compare against an ablation with the salience term removed.
• Robustness under model error: degrade the generative model deliberately and measure how gracefully the soft policy degrades compared with its hard-max twin.
• Preference-reward conversion: if the system switches between reward and preference descriptions, do the two produce consistent rankings of the same candidate policies?

• Control problems can be restated as inference over behavior: augment a bare generative model with binary optimality factors tied to reward through $p(\mathcal{O}_t{=}1) \propto \exp(r/\beta)$, using a state-action-independent omitted multiplier, and posterior conditioning gives $p(\tau \mid \mathcal{O}_{1:T}) \propto p(\tau)\, e^{R(\tau)/\beta}$ after those multipliers are absorbed into the global normalizer. This identity is the finite-horizon undiscounted construction; discounting requires time-indexed factors $e^{\gamma^{t-1}r_t/\beta}$, while the implementation uses the related discounted maximum-entropy Bellman surrogate.
• Exact backward messages in the augmented finite-horizon graph use a log-of-expectation state update. A related maximum-entropy Bellman surrogate replaces the hard action maximum with temperature-controlled log-sum-exp and induces a Boltzmann policy over soft Q-values.
• The same recursions optimize the maximum-entropy control objective, equipping policies with hedging behavior that hedges model error, a property model-based RL leans on heavily, as the next chapter explores in depth.
• When exact posteriors are unattainable, free energy supplies the universal variational currency: minimize $\mathcal{F}(q) = \mathbb{E}_q[-\log p(\text{model}, \text{evidence})] - \mathcal{H}[q]$ over tractable $q$.
• Active inference extends the currency prospectively: candidate policies are graded by expected free energy. Its risk-plus-ambiguity decomposition is equivalent to preference cross-entropy minus hidden-state information gain; uncertain-parameter formulations can add a novelty term for expected information gain about model parameters.
• The frameworks agree on softness, share softmax mathematics, and differ on where objectives come from, whether agents know their own models are incomplete, and how the exchange rate between knowing and getting is fixed. These are engineering degrees of freedom, and exercising them well is a recurring judgment call in the chapters ahead.

You now hold the tool that unifies the first five chapters of this part: one grammar in which filters, planners, and controllers are all posterior machines. Next we put pressure on that grammar. In Model-Based Reinforcement Learning, the generative model stops being known and becomes something learned from data, and we discover what remains of optimality when the evidence is produced by a network that can be confidently, catastrophically wrong.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about control as inference and active inference.