Control as Inference and Active Inference › Free Energy When Exact Inference Is Impossible

Everything so far assumed the trajectory posterior is computable. The finite tabular chain with known dynamics and one observed optimality factor per timestep retains a local factorization, so forward-backward message passing can compute its marginals exactly. What fails in realistic world models, with continuous state, partial observability, or learned nontrivial dynamics, is usually closed-form evaluation of the required sums or integrals, not the chain factorization itself. A different construction with one unobserved global tilt shared across timesteps can also destroy the per-step factorization. Our toy grid world uses a discounted maximum-entropy Bellman surrogate and samples its induced policy rather than drawing exact finite-horizon posterior trajectories. For these less tractable settings we approximate, and the canonical approximation device is variational inference, already your acquaintance from latent-variable modeling.

Pick a tractable family of distributions $q(\tau)$ and minimize the gap between $q$ and the true posterior. That gap is measured by a KL divergence we cannot evaluate directly, since evaluating it would require knowing the very posterior we are trying to approximate, but it is sandwiched by a quantity we can:

$\mathcal{F}(q)$ is the (negative) evidence lower bound, called free energy for reasons inherited from statistical physics: it trades off expected "energy" (how well $q$ fits the model and evidence) against entropy (how much $q$ refuses to commit). The physics analogy is more than decoration. In thermodynamics, systems at fixed temperature settle into states balancing energetic preference against entropic dispersion, and Helmholtz free energy measures exactly that compromise; variational inference borrows the name because it minimizes the same structural tradeoff, with $q$ playing the role of the system's configuration distribution. Minimizing $\mathcal{F}$ both tightens the bound on evidence and drives $q$ toward the true posterior; equality holds exactly when $q$ equals the posterior, in which case the bound saturates and the gap vanishes identically.

To make the tradeoff concrete, consider a miniature world whose true posterior over two outcomes is known, then let $q$ be a Bernoulli family indexed by its own probability parameter. Sweeping that parameter traces the full free-energy curve: its cross-entropy term falls as $q$ shifts mass toward the evidence-favored outcome, its negative-entropy term resists complete commitment, and their sum reaches its minimum precisely at the true posterior.

The seed for any sampling in the setup cells is fixed by the hidden seed_book_plots() call at the chapter opening, so the curves are identical on every render.

The dashed cross-entropy term alone is linear in $q$ and would drive it to the boundary $q=1$, placing all mass on the more probable outcome. The dotted negative-entropy term resists that complete commitment; their sum bottoms out at $q=0.75$, exactly where $q$ reproduces the true posterior. That is the graphical meaning of a saturated evidence bound. Readers who have met variational posteriors will recognize the shape immediately, since the same curve governs every ELBO ever maximized. Written out fully for trajectories, the quantity being minimized is

Every ingredient here is a friend from earlier chapters. The energy term scores proposed trajectories by exactly the generative model a world model provides; the entropy term preserves the hedging spirit of the previous section, ensuring the optimum does not collapse onto a single point estimate of the future. Because this machinery makes contact with your study of variational latent-variable models, note the kinship with the ELBO: same inequality, different cast of characters. There the evidence was a datapoint and $q$ approximated a posterior over latents; here the evidence is "the trajectory was near-optimal" and $q$ approximates a posterior over complete behaviors. The algebra does not notice the difference.

Applied to control specifically, the idea becomes powerful when the variational family $q$ is itself a set of feedback policies: if $q(\tau)$ factors as the law of the state process under a time-varying policy $\pi_t(a \mid s)$, minimizing free energy selects the best policy available in that family. This is optimization through an inference objective, not a guarantee of monotone Bellman policy improvement. An implementation may avoid writing an explicit value table, but the exact messages still satisfy the same soft Bellman structure derived above; the computational recursion has been reinterpreted, not abolished. Related inference and path-integral formulations yield tractable risk-sensitive linear-quadratic special cases, showing that the lens is not confined to grids and tables. In continuous spaces with Gaussian dynamics it can produce weighted-trajectory formulas; in learned models, trained as in Part V and Part VI, it becomes an optimizer over imagined rollouts, grading candidate futures against the generative model before any action touches the real world.

Keep one asymmetry in mind, because it is the hinge on which the next section swings. Free energy $\mathcal{F}$ is retrospective: it scores beliefs about trajectories that include evidence you have. It answers "how well did my beliefs explain what I observed?", a question about the past and present. An agent that must commit to future actions needs a prospective cousin, a quantity that scores proposed courses of action before the evidence arrives, averaging over the observations they would produce. That cousin, the expected free energy, is where active inference lives.