Active Perception, Dual Control, and Exploration › Information Gain and Uncertainty Reduction

Information gain is the expected reduction in uncertainty about a specified random variable that results from taking an action and observing its outcome. We will define it in the Shannon sense, derive its connection to the belief-MDP, and then compute it. Standard continuity, symmetry, and composition axioms single out Shannon entropy up to a choice of scale. Other uncertainty and risk measures remain useful under different objectives, so the information target and decision criterion must always be named.

For a discrete belief $b$ over a fixed finite state set, the entropy is

$$H(b) = -\sum_s b(s) \log b(s),$$

where:

• $b(s)$: the belief probability assigned to state $s$, with $b(s) \ge 0$ and $\sum_s b(s) = 1$
• $\log$: the natural logarithm, so entropy is measured in nats
• the sum runs over all states $s$ in the state space

with the convention $0 \log 0 = 0$. Entropy is nonnegative, zero exactly when the belief is a point mass, and maximized by the uniform distribution. It is measured in nats (natural logs) or bits (base-2 logs); the choice of base rescales entropy and information gain by a constant.

For a dynamic hidden state, the prior used for the next observation is not generally the current belief. After action $a$, the predictive belief over the next state is

$$b_a^-(s') = \sum_s T(s' \mid s,a)b(s).$$

The observation-conditioned posterior is $\tau(b,a,o)(s')$. Its entropy is

$$H(S' \mid O=o,b,a) = H\!\left(\tau(b, a, o)\right),$$

where $\tau$ is again the Bayesian update. The expected posterior entropy after taking action $a$ is the average over possible observations weighted by their predictive probability:

$$\mathbb{E}_{o \sim P(o \mid b, a)}\left[ H(S' \mid O=o,b,a) \right] = \sum_o P(o \mid b, a)\, H\!\left(\tau(b, a, o)\right).$$

This expression averages over what the observation could turn out to be. On the right, the sum weights each possible posterior entropy $H(\tau(b, a, o))$ by the probability $P(o \mid b, a)$ that the corresponding observation occurs. Once you have seen the observation, the posterior is deterministic, so there is no residual averaging inside a given branch; the averaging is entirely over which branch you land in.

The information gain about the next state from action $a$ is the mutual information between $S'$ and $O$, conditional on the current belief and action:

$$I_{S'}(b, a) = H(b_a^-) - \mathbb{E}_{o \sim P(o \mid b, a)}\left[ H\!\left(\tau(b,a,o)\right) \right].$$

where:

• $H(b_a^-)$: the entropy of the action-conditioned predictive prior over $S'$
• $P(o \mid b, a)$: the predictive probability of observing $o$ after taking action $a$ from belief $b$
• $H(\tau(b, a, o))$: the entropy of the posterior over $S'$ after observing $o$
• $\tau(b, a, o)$: the Bayesian posterior update
• $I_{S'}(b, a)$: the expected information gain, in nats, about the named target $S'$

For a static hidden target or parameter $X$, action does not transition $X$, so the predictive prior equals the current prior. The worked example later uses exactly this special case: $X$ is the identity of the rewarded target and $b_a^-=b$.

This quantity has three equivalent interpretations that are worth stating, because different literatures emphasize different ones:

• Entropy reduction: how much, on average, the observation reduces uncertainty about the named latent variable.
• Mutual information: $I(S';O\mid b,a)$, the dependence between predictive next state and observation for the chosen action.
• Expected posterior-to-predictive-prior KL: $I_{S'}(b,a)=\mathbb{E}_{o}[D_{\mathrm{KL}}(\tau(b,a,o)\|b_a^-)]$.

The third form is useful for Bayesian-surprise methods. Other curiosity bonuses introduced later are only surrogates for this quantity.

Write the expected KL from predictive prior to posterior. Expanding the KL gives

$$\mathbb{E}_{o}\!\left[ D_{\mathrm{KL}}\!\left( \tau(b,a,o) \,\|\, b_a^- \right) \right] = \sum_o P(o \mid b, a) \sum_{s'} \tau(b,a,o)(s') \log \frac{\tau(b,a,o)(s')}{b_a^-(s')}.$$

Substitute $\tau(b,a,o)(s')=O(o\mid s',a)b_a^-(s')/P(o\mid b,a)$ and rearrange:

$$\mathbb{E}_{o}\!\left[ D_{\mathrm{KL}}\!\left( \tau(b,a,o) \,\|\, b_a^- \right) \right] = H(b_a^-) - \mathbb{E}_{o}\!\left[ H\!\left( \tau(b,a,o) \right) \right] = I_{S'}(b, a).$$

This is the standard mutual-information identity for the joint distribution induced by the predictive prior and observation channel. The prior and posterior must concern the same random variable; comparing a posterior over $S'$ with a pre-action belief over $S$ is not meaningful in general.

Reading the algebra backward is useful, with one qualification. Maximizing expected posterior-to-predictive-prior KL maximizes information gain about the named variable. Minimizing expected posterior entropy is equivalent only when predictive-prior entropy is fixed across the actions being compared. Bayesian-surprise methods use this identity directly; count bonuses, prediction error, RND, and ensemble disagreement are different surrogates and need not estimate the same quantity.

A useful sanity property: $I_{S'}(b, a) \ge 0$ for every belief and action, with equality if and only if $O$ and $S'$ are conditionally independent for that $(b,a)$. This follows directly from $D_{\mathrm{KL}} \ge 0$. A negative numerical estimate therefore reflects approximation, sampling, or floating-point error rather than negative exact mutual information.

Information gain is also invariant to relabeling of the observations: it is built from posterior entropies, which do not depend on the symbols assigned to those posteriors. If you permute the observation symbols and the observation model consistently, information gain does not move. An action that reliably reveals "not A" is just as informative as one that reliably reveals "A": what matters is how sharply the observation is correlated with the latent state, not which symbol encodes it.

Look back at the belief-MDP recursion. The term $P(o \mid b, a)$ is the branch probability, and the downstream value $V^*(\tau(b,a,o))$ depends on the posterior. Information gain measures the expected posterior-to-predictive-prior KL induced by these branches. The weight $\beta$ converts information into objective units, but it cannot by itself determine whether the information changes a useful decision.

Not all information is equally valuable. The quantity relevant to control is not entropy in general but entropy that affects the optimal action. If two states are equally good under every action, distinguishing them has no value. This is the "value of information" perspective. It motivates a refinement of plain entropy.

The inner maximum chooses the best task decision after seeing $o$, whereas entropy-based information gain does not ask whether the posterior changes a useful choice. Resolving an irrelevant distinction changes the belief but leaves the maximum unchanged. This mismatch reappears in curiosity, where surprise-driven objectives can chase irrelevant novelty, and in safety, where useful probes may be inadmissible.

For continuous latent states, the exact belief is rarely available. Differential entropy depends on coordinates and may be negative or divergent; mutual information, when well defined, is invariant under smooth invertible reparameterizations. Practical systems therefore estimate mutual information directly or use a calibrated parametric or ensemble surrogate.

For a nonsingular Gaussian belief $p(s) = \mathcal{N}(s; \mu, \Sigma)$ over a $d$-dimensional continuous latent state, with positive-definite covariance $\Sigma$, the differential entropy is

$$H = \frac{1}{2} \log\!\left( (2\pi e)^d \det \Sigma \right) = \frac{d}{2} + \frac{d}{2} \log(2\pi) + \frac{1}{2} \log \det \Sigma,$$

where:

• $d$: the dimension of the latent state
• $\mu$: the mean of the Gaussian belief
• $\Sigma$: the covariance matrix encoding the belief's uncertainty
• $\det \Sigma$: the determinant of the covariance; $\sqrt{\det\Sigma}$ is proportional to the volume scale of a fixed Mahalanobis-radius ellipsoid
• $e$: Euler's number, arising from the maximum-entropy property of the Gaussian

Three implementations recur throughout this book:

• Ensemble disagreement: train $K$ diverse models and use their predictive spread as an epistemic proxy. That interpretation requires posterior-like diversity, calibration, and a separate representation of aleatoric output noise. Shared data alone does not make disagreement epistemic. PETS, for example, combines ensemble variation with probabilistic member outputs.
• Parametric differential entropy: for nonsingular Gaussian predictive and posterior beliefs, $$I=\tfrac12\left[\log\det\Sigma_{\text{prior}}-\mathbb{E}_o\log\det\Sigma_{\text{post}}(o)\right].$$ In a linear-Gaussian update the posterior covariance is independent of the realized observation, so this simplifies to one half the log-determinant ratio.
• Variational information estimates: optimize a learned lower or upper bound on the expected log density ratio, often through a conditional density or contrastive classifier. A raw target-class probability is not itself mutual information.

These are useful approximations, but the executable example below deliberately uses an exact two-state categorical belief so every posterior and information value can be checked directly. First, we sharpen the notion of curiosity, which is the motivational layer on top of information gain.